BERTnesia: Investigating the capture and forgetting of knowledge in BERT
Wallat, Jonas et al. “BERTnesia: Investigating the capture and forgetting of knowledge in BERT.” BLACKBOXNLP (2020).
CV
Education
- B.S. in Computer Science, Leibniz University Hannover, 2017
- M.S. in Computer Science, Leibniz University Hannover, 2020
- Ph.D in Interpretable AI, Leibniz University Hannover, 2024/2025 (expected)
Work experience
- October 2020-now: Researcher/PhD Student
- L3S Research Center Hannover
- Tasks: Research, teaching, student supervision, proposal writing
- Supervisors: Prof. Avishek Anand (TU Delft), Prof. Wolfgang Nejdl (L3S, Uni Hannover)
- Summer 2019: Software Development Intern
- OpenMarket, London
- Tasks included: Enterprise Java and Javascript development (AWS/Terraform)
- Supervisor: Geeta Kholia
- 2018-2019 (13 months, part-time): Student Trainee - Software Development
- PPI AG, Hannover
- Tasks included: Enterprise Java Development
- Supervisor: Dr. Karl-Ingo Friese
- 2016-2018 (18 months, part-time): Research Assistant
- Department of Software Engineering, Leibniz University Hannover
- Tasks included: Investigating communication behavior and its effects on project success
- Supervisor: Dr. Jil Klünder
Skills
- Python
- Pytorch
- Pytorch Lightning
- Transformers
- Weights and Biases
- Java
- Flutter & Dart
Publications
Explainable Information Retrieval: A Survey
Anand, Avishek, et al. "Explainable Information Retrieval: A Survey." arXiv preprint arXiv:2211.02405 (2022).
A Review of the Role of Causality in Developing Trustworthy AI Systems
Ganguly, Niloy, et al. "A review of the role of causality in developing trustworthy ai systems." arXiv preprint arXiv:2302.06975 (2023).
Probing BERT for Ranking Abilities
Wallat, Jonas, et al. "Probing BERT for ranking abilities." Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2–6, 2023, Proceedings, Part II. Cham: Springer Nature Switzerland, 2023.
The Effect of Masking Strategies on Knowledge Retention by Language Models
Wallat, J., Zhang, T., & Anand, A. (2023). The Effect of Masking Strategies on Knowledge Retention by Language Models. ArXiv, abs/2306.07185.
GeneMask: Fast Pretraining of Gene Sequences to Enable Few-Shot Learning
Roy, S., Wallat, J., Sundaram, S.S., Nejdl, W., & Ganguly, N. (2023). GeneMask: Fast Pretraining of Gene Sequences to Enable Few-Shot Learning. European Conference on Artificial Intelligence.