how to write a research proposal for llm

How To Write A Research Proposal

A Straightforward How-To Guide (With Examples)

By: Derek Jansen (MBA) | Reviewed By: Dr. Eunice Rautenbach | August 2019 (Updated April 2023)

Writing up a strong research proposal for a dissertation or thesis is much like a marriage proposal. It’s a task that calls on you to win somebody over and persuade them that what you’re planning is a great idea. An idea they’re happy to say ‘yes’ to. This means that your dissertation proposal needs to be   persuasive ,   attractive   and well-planned. In this post, I’ll show you how to write a winning dissertation proposal, from scratch.

Before you start:

– Understand exactly what a research proposal is – Ask yourself these 4 questions

The 5 essential ingredients:

• The title/topic
• The introduction chapter
• The scope/delimitations
• Preliminary literature review
• Design/ methodology
• Practical considerations and risks 

What Is A Research Proposal?

The research proposal is literally that: a written document that communicates what you propose to research, in a concise format. It’s where you put all that stuff that’s spinning around in your head down on to paper, in a logical, convincing fashion.

Convincing   is the keyword here, as your research proposal needs to convince the assessor that your research is   clearly articulated   (i.e., a clear research question) ,   worth doing   (i.e., is unique and valuable enough to justify the effort), and   doable   within the restrictions you’ll face (time limits, budget, skill limits, etc.). If your proposal does not address these three criteria, your research won’t be approved, no matter how “exciting” the research idea might be.

PS – if you’re completely new to proposal writing, we’ve got a detailed walkthrough video covering two successful research proposals here . 

How do I know I’m ready?

Before starting the writing process, you need to   ask yourself 4 important questions .  If you can’t answer them succinctly and confidently, you’re not ready – you need to go back and think more deeply about your dissertation topic .

You should be able to answer the following 4 questions before starting your dissertation or thesis research proposal:

• WHAT is my main research question? (the topic)
• WHO cares and why is this important? (the justification)
• WHAT data would I need to answer this question, and how will I analyse it? (the research design)
• HOW will I manage the completion of this research, within the given timelines? (project and risk management)

If you can’t answer these questions clearly and concisely,   you’re not yet ready   to write your research proposal – revisit our   post on choosing a topic .

If you can, that’s great – it’s time to start writing up your dissertation proposal. Next, I’ll discuss what needs to go into your research proposal, and how to structure it all into an intuitive, convincing document with a linear narrative.

The 5 Essential Ingredients

Research proposals can vary in style between institutions and disciplines, but here I’ll share with you a   handy 5-section structure   you can use. These 5 sections directly address the core questions we spoke about earlier, ensuring that you present a convincing proposal. If your institution already provides a proposal template, there will likely be substantial overlap with this, so you’ll still get value from reading on.

For each section discussed below, make sure you use headers and sub-headers (ideally, numbered headers) to help the reader navigate through your document, and to support them when they need to revisit a previous section. Don’t just present an endless wall of text, paragraph after paragraph after paragraph…

Top Tip:   Use MS Word Styles to format headings. This will allow you to be clear about whether a sub-heading is level 2, 3, or 4. Additionally, you can view your document in ‘outline view’ which will show you only your headings. This makes it much easier to check your structure, shift things around and make decisions about where a section needs to sit. You can also generate a 100% accurate table of contents using Word’s automatic functionality.

Ingredient #1 – Topic/Title Header

Your research proposal’s title should be your main research question in its simplest form, possibly with a sub-heading providing basic details on the specifics of the study. For example:

“Compliance with equality legislation in the charity sector: a study of the ‘reasonable adjustments’ made in three London care homes”

As you can see, this title provides a clear indication of what the research is about, in broad terms. It paints a high-level picture for the first-time reader, which gives them a taste of what to expect.   Always aim for a clear, concise title . Don’t feel the need to capture every detail of your research in your title – your proposal will fill in the gaps.

Need a helping hand?

Ingredient #2 – Introduction

In this section of your research proposal, you’ll expand on what you’ve communicated in the title, by providing a few paragraphs which offer more detail about your research topic. Importantly, the focus here is the   topic   – what will you research and why is that worth researching? This is not the place to discuss methodology, practicalities, etc. – you’ll do that later.

You should cover the following:

• An overview of the   broad area   you’ll be researching – introduce the reader to key concepts and language
• An explanation of the   specific (narrower) area   you’ll be focusing, and why you’ll be focusing there
• Your research   aims   and   objectives
• Your   research question (s) and sub-questions (if applicable)

Importantly, you should aim to use short sentences and plain language – don’t babble on with extensive jargon, acronyms and complex language. Assume that the reader is an intelligent layman – not a subject area specialist (even if they are). Remember that the   best writing is writing that can be easily understood   and digested. Keep it simple.

Note that some universities may want some extra bits and pieces in your introduction section. For example, personal development objectives, a structural outline, etc. Check your brief to see if there are any other details they expect in your proposal, and make sure you find a place for these.

Ingredient #3 – Scope

Next, you’ll need to specify what the scope of your research will be – this is also known as the delimitations . In other words, you need to make it clear what you will be covering and, more importantly, what you won’t be covering in your research. Simply put, this is about ring fencing your research topic so that you have a laser-sharp focus.

All too often, students feel the need to go broad and try to address as many issues as possible, in the interest of producing comprehensive research. Whilst this is admirable, it’s a mistake. By tightly refining your scope, you’ll enable yourself to   go deep   with your research, which is what you need to earn good marks. If your scope is too broad, you’re likely going to land up with superficial research (which won’t earn marks), so don’t be afraid to narrow things down.

Ingredient #4 – Literature Review

In this section of your research proposal, you need to provide a (relatively) brief discussion of the existing literature. Naturally, this will not be as comprehensive as the literature review in your actual dissertation, but it will lay the foundation for that. In fact, if you put in the effort at this stage, you’ll make your life a lot easier when it’s time to write your actual literature review chapter.

There are a few things you need to achieve in this section:

• Demonstrate that you’ve done your reading and are   familiar with the current state of the research   in your topic area.
• Show that   there’s a clear gap   for your specific research – i.e., show that your topic is sufficiently unique and will add value to the existing research.
• Show how the existing research has shaped your thinking regarding   research design . For example, you might use scales or questionnaires from previous studies.

When you write up your literature review, keep these three objectives front of mind, especially number two (revealing the gap in the literature), so that your literature review has a   clear purpose and direction . Everything you write should be contributing towards one (or more) of these objectives in some way. If it doesn’t, you need to ask yourself whether it’s truly needed.

Top Tip:  Don’t fall into the trap of just describing the main pieces of literature, for example, “A says this, B says that, C also says that…” and so on. Merely describing the literature provides no value. Instead, you need to   synthesise   it, and use it to address the three objectives above.

Ingredient #5 – Research Methodology

Now that you’ve clearly explained both your intended research topic (in the introduction) and the existing research it will draw on (in the literature review section), it’s time to get practical and explain exactly how you’ll be carrying out your own research. In other words, your research methodology.

In this section, you’ll need to   answer two critical questions :

• How   will you design your research? I.e., what research methodology will you adopt, what will your sample be, how will you collect data, etc.
• Why   have you chosen this design? I.e., why does this approach suit your specific research aims, objectives and questions?

In other words, this is not just about explaining WHAT you’ll be doing, it’s also about explaining WHY. In fact, the   justification is the most important part , because that justification is how you demonstrate a good understanding of research design (which is what assessors want to see).

Some essential design choices you need to cover in your research proposal include:

• Your intended research philosophy (e.g., positivism, interpretivism or pragmatism )
• What methodological approach you’ll be taking (e.g., qualitative , quantitative or mixed )
• The details of your sample (e.g., sample size, who they are, who they represent, etc.)
• What data you plan to collect (i.e. data about what, in what form?)
• How you plan to collect it (e.g., surveys , interviews , focus groups, etc.)
• How you plan to analyse it (e.g., regression analysis, thematic analysis , etc.)
• Ethical adherence (i.e., does this research satisfy all ethical requirements of your institution, or does it need further approval?)

This list is not exhaustive – these are just some core attributes of research design. Check with your institution what level of detail they expect. The “ research onion ” by Saunders et al (2009) provides a good summary of the various design choices you ultimately need to make – you can   read more about that here .

Don’t forget the practicalities…

In addition to the technical aspects, you will need to address the   practical   side of the project. In other words, you need to explain   what resources you’ll need   (e.g., time, money, access to equipment or software, etc.) and how you intend to secure these resources. You need to show that your project is feasible, so any “make or break” type resources need to already be secured. The success or failure of your project cannot depend on some resource which you’re not yet sure you have access to.

Another part of the practicalities discussion is   project and risk management . In other words, you need to show that you have a clear project plan to tackle your research with. Some key questions to address:

• What are the timelines for each phase of your project?
• Are the time allocations reasonable?
• What happens if something takes longer than anticipated (risk management)?
• What happens if you don’t get the response rate you expect?

A good way to demonstrate that you’ve thought this through is to include a Gantt chart and a risk register (in the appendix if word count is a problem). With these two tools, you can show that you’ve got a clear, feasible plan, and you’ve thought about and accounted for the potential risks.

Tip – Be honest about the potential difficulties – but show that you are anticipating solutions and workarounds. This is much more impressive to an assessor than an unrealistically optimistic proposal which does not anticipate any challenges whatsoever.

Final Touches: Read And Simplify

The final step is to edit and proofread your proposal – very carefully. It sounds obvious, but all too often poor editing and proofreading ruin a good proposal. Nothing is more off-putting for an assessor than a poorly edited, typo-strewn document. It sends the message that you either do not pay attention to detail, or just don’t care. Neither of these are good messages. Put the effort into editing and proofreading your proposal (or pay someone to do it for you) – it will pay dividends.

When you’re editing, watch out for ‘academese’. Many students can speak simply, passionately and clearly about their dissertation topic – but become incomprehensible the moment they turn the laptop on. You are not required to write in any kind of special, formal, complex language when you write academic work. Sure, there may be technical terms, jargon specific to your discipline, shorthand terms and so on. But, apart from those,   keep your written language very close to natural spoken language   – just as you would speak in the classroom. Imagine that you are explaining your project plans to your classmates or a family member. Remember, write for the intelligent layman, not the subject matter experts. Plain-language, concise writing is what wins hearts and minds – and marks!

Let’s Recap: Research Proposal 101

And there you have it – how to write your dissertation or thesis research proposal, from the title page to the final proof. Here’s a quick recap of the key takeaways:

• The purpose of the research proposal is to   convince   – therefore, you need to make a clear, concise argument of why your research is both worth doing and doable.
• Make sure you can ask the critical what, who, and how questions of your research   before   you put pen to paper.
• Title – provides the first taste of your research, in broad terms
• Introduction – explains what you’ll be researching in more detail
• Scope – explains the boundaries of your research
• Literature review – explains how your research fits into the existing research and why it’s unique and valuable
• Research methodology – explains and justifies how you will carry out your own research

Hopefully, this post has helped you better understand how to write up a winning research proposal. If you enjoyed it, be sure to check out the rest of the Grad Coach Blog . If your university doesn’t provide any template for your proposal, you might want to try out our free research proposal template .

Psst… there’s more!

This post is an extract from our bestselling short course, Research Proposal Bootcamp . If you want to work smart, you don't want to miss this .

30 Comments

Thank you so much for the valuable insight that you have given, especially on the research proposal. That is what I have managed to cover. I still need to go back to the other parts as I got disturbed while still listening to Derek’s audio on you-tube. I am inspired. I will definitely continue with Grad-coach guidance on You-tube.

Thanks for the kind words :). All the best with your proposal.

First of all, thanks a lot for making such a wonderful presentation. The video was really useful and gave me a very clear insight of how a research proposal has to be written. I shall try implementing these ideas in my RP.

Once again, I thank you for this content.

I found reading your outline on writing research proposal very beneficial. I wish there was a way of submitting my draft proposal to you guys for critiquing before I submit to the institution.

Hi Bonginkosi

Thank you for the kind words. Yes, we do provide a review service. The best starting point is to have a chat with one of our coaches here: https://gradcoach.com/book/new/ .

Hello team GRADCOACH, may God bless you so much. I was totally green in research. Am so happy for your free superb tutorials and resources. Once again thank you so much Derek and his team.

You’re welcome, Erick. Good luck with your research proposal 🙂

thank you for the information. its precise and on point.

Really a remarkable piece of writing and great source of guidance for the researchers. GOD BLESS YOU for your guidance. Regards

Thanks so much for your guidance. It is easy and comprehensive the way you explain the steps for a winning research proposal.

Thank you guys so much for the rich post. I enjoyed and learn from every word in it. My problem now is how to get into your platform wherein I can always seek help on things related to my research work ? Secondly, I wish to find out if there is a way I can send my tentative proposal to you guys for examination before I take to my supervisor Once again thanks very much for the insights

Thanks for your kind words, Desire.

If you are based in a country where Grad Coach’s paid services are available, you can book a consultation by clicking the “Book” button in the top right.

Best of luck with your studies.

May God bless you team for the wonderful work you are doing,

If I have a topic, Can I submit it to you so that you can draft a proposal for me?? As I am expecting to go for masters degree in the near future.

Thanks for your comment. We definitely cannot draft a proposal for you, as that would constitute academic misconduct. The proposal needs to be your own work. We can coach you through the process, but it needs to be your own work and your own writing.

Best of luck with your research!

I found a lot of many essential concepts from your material. it is real a road map to write a research proposal. so thanks a lot. If there is any update material on your hand on MBA please forward to me.

GradCoach is a professional website that presents support and helps for MBA student like me through the useful online information on the page and with my 1-on-1 online coaching with the amazing and professional PhD Kerryen.

Thank you Kerryen so much for the support and help 🙂

I really recommend dealing with such a reliable services provider like Gradcoah and a coach like Kerryen.

Hi, Am happy for your service and effort to help students and researchers, Please, i have been given an assignment on research for strategic development, the task one is to formulate a research proposal to support the strategic development of a business area, my issue here is how to go about it, especially the topic or title and introduction. Please, i would like to know if you could help me and how much is the charge.

This content is practical, valuable, and just great!

Thank you very much!

Hi Derek, Thank you for the valuable presentation. It is very helpful especially for beginners like me. I am just starting my PhD.

This is quite instructive and research proposal made simple. Can I have a research proposal template?

Great! Thanks for rescuing me, because I had no former knowledge in this topic. But with this piece of information, I am now secured. Thank you once more.

I enjoyed listening to your video on how to write a proposal. I think I will be able to write a winning proposal with your advice. I wish you were to be my supervisor.

Dear Derek Jansen,

Thank you for your great content. I couldn’t learn these topics in MBA, but now I learned from GradCoach. Really appreciate your efforts….

From Afghanistan!

I have got very essential inputs for startup of my dissertation proposal. Well organized properly communicated with video presentation. Thank you for the presentation.

Wow, this is absolutely amazing guys. Thank you so much for the fruitful presentation, you’ve made my research much easier.

this helps me a lot. thank you all so much for impacting in us. may god richly bless you all

How I wish I’d learn about Grad Coach earlier. I’ve been stumbling around writing and rewriting! Now I have concise clear directions on how to put this thing together. Thank you!

Fantastic!! Thank You for this very concise yet comprehensive guidance.

Even if I am poor in English I would like to thank you very much.

Thank you very much, this is very insightful.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Support NYU Law

• Graduate Affairs
• Research & Writing

LLM Research and Writing Options

Working as a research assistant for a law school professor.

Faculty members may offer students the opportunity to work as research assistants (RAs) for monetary compensation or, if the professor deems it appropriate based on the nature of the work, for academic credit.  For details, review information on serving as a research assistant for faculty .

Directed Research

 To undertake Directed Research, students contact individual instructors and agree on a research project. To register, a written proposal must be approved and signed by the instructor, and then submitted to the Office of Graduate Affairs. The written proposal should be at least 1000 words and describe the subject matter of the Directed Research and the issues the student intends to explore in the paper.  While any full-time faculty member or visiting faculty member may supervise the research, Adjunct Professors may supervise only with the permission of Vice Dean Hertz.

Directed Research credit may be added through Monday, September 30 for Fall 2024, and Monday, February 3 for Spring 2025.

The usual allocation for Directed Research is two credits. A student may write a one-credit Directed Research.  A two-credit Directed Research project should conform to the r equirements for an Option A paper ; a one-credit Directed Research paper should be at least 5,000 words, exclusive of footnotes. A three-credit Directed Research project is highly unusual and requires the approval of Vice Dean Randy Hertz.  Students considering a 3-credit Directed Research should contact the Office of Graduate Affairs to discuss. 

For non-tax students no more than four of a student's 24 credits may consist of directed research. Tax students may take a maximum of two credits of directed research. Regardless of the type of project involved, students are, of course, expected to submit original, non-duplicative work. When in doubt about proper use of a citation or quotation, discuss the issue with the instructor. Plagiarism is a serious offense that may merit severe discipline. Requests to add Directed Research after the deadline stated above require approval of Vice Dean Hertz. Such requests should be initiated by contacting the Office of Graduate Affairs and will only be considered if your credit load (not including the Directed Research credits) does not drop below minimum requirements after the add/drop period. Students who are granted permission to late-add Directed Research will not be permitted to drop courses if the result is inconsistent with the above; please plan your schedule accordingly. After March 15, the Vice Dean may allow a student to add Directed Research only in exceptional circumstances. No more than two credits can be earned in this manner.

Read further about Requirements for Directed Research

Directed Research During the Summer Semester

Students may register for Directed Research during the summer semester. The summer registration deadlines is July 1, unless there is approval by the Vice Dean to add at a later date. Please note that full-time students will be charged per credit for Directed Research during the summer. All work must be submitted by September 1 or by an earlier deadline established by the supervising faculty member.

Writing Credit

In seminars, colloquia, and courses that offer the option to add an additional writing credit, students may earn one credit for writing a substantial paper (at least 10,000 words in length exclusive of footnotes). To earn the additional credit, students must register for the writing credit section of the course within the same semester the course is offered. The deadline for registering is Monday, September 30 for Fall 2024, and Monday, February 3 for Spring 2025.

LLM Thesis Option

LLM students have the option to write a substantial research paper, in conjunction with a seminar or Directed Research that may be recorded as a "thesis" on their transcript. At the onset of the seminar or Directed Research, the student must obtain approval from the professor that the paper will be completed for a "thesis" designation.

It should be substantial in length (at least 10,000 words exclusive of footnotes) and, like the substantial writing requirement for JD students, must be analytical rather than descriptive in nature, showing original thought and analysis. Please note the thesis designation is for a single research paper agreed upon in advance.

The student is required to submit an outline and at least one FULL PRE-FINAL draft to the faculty member in order to receive the thesis notation. When submitting a final draft of the thesis to the faculty member, the student must give the faculty member an LLM Thesis Certification form . The faculty member is required to return the signed form to the Office of Records and Registration when submitting a grade for the course.

Please note that the student will not receive additional credit for writing the thesis, but will only receive credit for the seminar or Directed Research for which he or she is registered.

International Legal Studies Students should review their program requirements for further information about writing an LLM thesis within their program.

Writing Assistance

Writing resources.

• Guide to Writing
• (excellent guide to legal writing generally)
• So You Want to Write a Research Paper...
• (Recording with Prof. Jose Alvarez)
• So You Want to Write About International Law...
• Some Thoughts on Writing by Barry Friedman (PDF: 106 KB)
• NYU Law Library Guide: Researching and Writing a Law Review Note or Seminar Paper
• NYU Law Library Research Guides
• Why Write a Student Note

© 2024 New York University School of Law. 40 Washington Sq. South, New York, NY 10012.   Tel. (212) 998-6100

• Skip to main content

We use cookies

Necessary cookies.

Necessary cookies enable core functionality. The website cannot function properly without these cookies, and can only be disabled by changing your browser preferences.

Analytics cookies

Analytical cookies help us improve our website. We use Google Analytics. All data is anonymised.

Clarity helps us to understand our users’ behaviour by visually representing their clicks, taps and scrolling. All data is anonymised.

Privacy policy

• School of Law

Writing a research proposal

• Our research students

Preparing your research proposal is the important first step to becoming a postgraduate research student at the School of Law.

The focus of your proposal will be slightly different depending on whether you wish to do a PhD or an LLM by research, but the principles of what to include and who to contact for advice are the same.

Speaking to a potential supervisor

Before you write your detailed research proposal, you may wish to contact a member of  our research staff  with knowledge of the subject area. They who should be able to advise you whether or not your proposed topic is feasible. 

This can be done prior to a formal application. 

If you are not sure who is the best person to contact, an initial enquiry can be made to our Postgraduate Administrator,  Susan Holmes .

What to include in your proposal

A proposal for an LLM by research or a PhD should not exceed 15 pages in length and is unlikely to be less than 8 pages in length.

Check the limit specified by the funding body to which you are applying.

It should include the following:

A working title

The research context.

This is the background against which your research will be carried out.

It should be a brief introduction outlining the general area of study and identifying the subject area within which your study falls. You should also refer to the current state of knowledge (i.e. what research has been done to date) and any recent debates on the subject.

You need to reference this in the same way as you would do if you were writing an essay e.g. any articles or books you refer to should have a footnote with the full details of author, title, publication date, etc.

The research issue, aims or questions

Outline the contribution that your research will make. It is normally best to do this in the form of specific aims or research questions or issues.

The importance of your proposed research

Demonstrate how your research fills a gap in existing research, by showing that it hasn’t been done before.

Explain why your research is important. It is not enough to say that this has not been studied previously, you need to explain why it is important or interesting enough to be studied.

‎Research methods

Here you need to explain how you will obtain the information necessary to write your thesis.

• Explain whether you will use secondary and/or primary sources
• Give some detail on exactly how you will obtain your information

For most law students, you will probably rely on documentary sources – information that already exists in some form e.g. journal articles, case reports, legislation, treaties, historical records.

In this case you need to say a little about how you will access these (bearing in mind that as a student of the University you will be provided with access to legal databases including Westlaw and LexisLibrary).

If yours is a comparative or international study, you will need to explain how you will obtain the relevant international materials and whether or not this will involve travel.

Some studies, however, might involve empirical research – information that is gathered through direct interaction with people and processes such as interviews, questionnaires, court observation or analysis of private records.

If you plan to undertake empirical research, you need to explain why this is an appropriate research method and give details of your planned methodology (e.g. who you hope to interview, how many interviews you will carry out).

In this section, you should also explain any special skills you have that will assist you in obtaining information, for example, if you plan to look at French law and you can read or speak French.

You should provide a very approximate timetable for the research.

For example, the timetable for a research LLM thesis comparing French law and Scots law might be:

• months 1-3 reading theoretical material and developing theoretical framework
• months 4-6 reading and analysing French materials
• months 7-9 reading and analysing Scottish materials
• months 9-12 writing up the thesis

Research proposals for a PhD

When choosing a subject for your thesis, consider the requirements for a relevant degree and whether you can stick within the time and word limits. A PhD thesis must be from 70,000 to 100,000 words including footnotes.

Consider how your study will demonstrate originality. It is not enough simply to reproduce existing knowledge. There are many ways in which you can do this – it does not necessarily require you to study something that has never been studied before in any way, shape or form. For example, you could:

• Study something that has never been studied before
• Bring new insights to an existing area of legal thought
• Work between disciplines eg. by applying philosophical, psychological or sociological analysis to legal issues
• Bring together areas of legal thought that have not been brought together before eg. use concepts from property law to analyse sexual offences
• Analyse new case law/new legislation in a particular area of law
• Identify new problems with existing case law/legislation in a particular area of law
• Undertake an empirical study to see if the law is achieving its objectives

You also need to make sure your topic is not too broad.  It is inappropriate to write a thesis that reads like a textbook.  This is not sufficiently advanced work and your treatment will be too superficial.  You need to choose something that will give you the scope both to describe and critically analyse the law.  For example, a thesis on “the law relating to criminal defences inScotland” or “a review of EC law governing the enforcement of European law in national courts of member states” would be too broad.  You would have to narrow down your topic to consideration of one particular aspect of the topic (e.g. one specific defence or one specific aspect of European law).

Recent and current PhD thesis topics have included: 

• Peacekeepers as enforcers? A legal analysis of the attribution of enforcement powers to UN peacekeeping operations in the new millenium
• The impact of the World Trade Organisation on the formulation of the anti-monopoly law of the People’s Republic ofChina
• Access to employment and career progression for women in the European labour market
• Consent to medical treatment and the competent adult
• Migratory things on or beneath land: a study of property and rights of use
• The effect of the constitutional relations betweenScotlandandEnglandon their conflict of laws relations: a Scottish perspective
• Persuasion: a historical-comparative study of the role of persuasion within the judicial decision-making process
• Law reform proposals for the protection of the right to seek refugee status in the European Community
• Historicizing the criminalization of youth

Research proposals for an LLM by research

For an LLM by research, your study should still be critical rather than simply describing the law in a particular area.

The field of study is likely to be significantly narrower than for a PhD, as it has a 30,000 word limit.

Recent and current LLM by research thesis topics have included:

• Sustainable development and urban governance in planning law
• Domestic abuse and Scots law
• Criminal liability for individuals who fail to prevent harm
• Legal and scientific evidence of torture
• The responsibility of international organisations: efforts of the international law commission

• Home »
• Editorial »
• Advice »
• LLM Student Life »

Guide To LLM Dissertation Writing

Find your perfect llm program search our database of over 2500 courses.

Choosing the topic

Five key considerations when choosing your dissertation topic are:

• Why are you studying an LLM in thr first place?
• Which modules have you enjoyed the most?
• Which areas of the law have a good support base at your institution?
• What are the strengths of your law library?
• Is your potential topic a wide enough question?

Obviously, choosing a topic is a necessary stage to get underway before you can start researching and writing your dissertation. You should spend time carefully considering the subject of your dissertation as it might end up being the clincher for that  first job  after you finish your studies. Make sure you pick a topic that you find interesting, but that also has the balance of support from your lecturers and professors.

You'll need to be uniquely motivated to produce a dissertation about a subject that none of your lectures or professors knows much about. Understanding why you have picked the particular subject will ensure you're choosing the right topic, but don't spend too much time considering what to do as you'll need to get on with it. 

Dissertation support

You will have plenty of dissertation support organised through your law school. Some of it might be compulsory sessions that you must undertake as part of your LLM program, while others may be support sessions that can help you stay focused throughout your dissertation work.

Make sure you attend these sessions and don’t hesitate to ask questions if in doubt. It might be a good idea to share your dissertation structure with tutors or designated academic contacts that can give you feedback on your progress. Law school libraries usually have  books  that tell you how best to prepare for your dissertation. Keep an eye out for skills sessions on writing or research methods. These will prove useful when you get down to drafting content for your dissertation and will enable you to put to practice acquired skills that you picked up during these sessions.

Planning and organisation

Some people love creating a filing system and hopefully, you're one of them as this is a great way to organise your LLM dissertation. You'll need to keep your research well organised to enable you to quickly access it when you are writing your dissertation. It's a good idea to have research divided into chapters early on.

It's a good idea to follow a file management procedure to save your dissertation material. This material could consist of both printed (photocopies from the library or print-outs of research articles) and online documents. Try to follow a consistent labelling/naming convention so that you can locate documents quickly. For instance, if you have a vast number of online articles and research papers to go through, then categorise them in such a way that they fall under relevant chapters of your dissertation.

Any research you do online will need to be backed up, and of course, you will have the dissertation itself backed up too. Do not have everything saved on one ancient laptop, instead build in a routine for how you save and backup your data daily so it just becomes part of how you work. If you start as early as you can on your dissertation, then you'll be able to build in planning time and create a realistic timetable for your work, with escapes from your dissertation to let you reflect on what you have done so far. 

Researching

There is no easy or quick way around this, you are just going to have to get going with the research as soon as possible. Remember that law libraries get busy during the second semester so you'll need to get there early in the mornings or stay late sometimes.

You also don't want to wait around for particular texts that have a limited availability. If you realise you need a book that someone else has checked-out of the library, then let the staff know as soon as you know so you have a chance of getting it. Don't forget about online law libraries and resources too, and speak with your academic staff if you are really struggling to access what you need. 

Don't wait until you think you've done all the researching before you start writing up your findings. Writing up an LLM dissertation takes time and thought. Start writing as soon as you start researching and keep planning the chapters of your dissertation as you delve deeper into the research. With a bit of luck and good planning, you will find that the chapters are easy to write. 

Editing and formatting

Find out before you write a single word what format your dissertation needs to be in for printing and submission. Your law school will likely have their own standards, so you should familiarise yourself with this document before you get started. Establish the right format straight away so you are not spending time at the last minute changing formats or the way you have referenced the whole document.

Don't underestimate how long it will take to edit your dissertation – expect to read through each chapter many times as each read through will show you new and interesting mistakes. And if possible, find a willing friend or family member to give it a final read – fresh eyes are likely to pick up small typos or mistakes.  

Printing and submitting

Aim to finish your LLM dissertation with a little time to spare. Towards submission deadlines, university printers are busy places and if you need to print your dissertation at a particular printer then check with them early on to understand how much time they need to get your document ready. You need time for them to print it and time for you to check the printed material as you need to check for formatting errors or any printing mistakes like double pages.  Once you've written your dissertation you can take a calm walk into your submissions office and hand over your dissertation. Obviously, everything went to plan and you've finished your dissertation with time to spare and now it's time to  relax a little .

Related articles

Global LLM Study Bursaries

• Law bursaries
• Open day alerts
• Funding advice
• Application tips
• Law & LLM news

Complete Our Destination Survey!

Take 2 minutes to complete our Destination Survey for the chance to win a £2,000 LLM & PG Law Bursary.

All we need to know is:

• Your university or law school
• Your PG law course

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation

How to Write a Dissertation Proposal | A Step-by-Step Guide

Published on 14 February 2020 by Jack Caulfield . Revised on 11 November 2022.

A dissertation proposal describes the research you want to do: what it’s about, how you’ll conduct it, and why it’s worthwhile. You will probably have to write a proposal before starting your dissertation as an undergraduate or postgraduate student.

A dissertation proposal should generally include:

• An introduction to your topic and aims
• A literature review  of the current state of knowledge
• An outline of your proposed methodology
• A discussion of the possible implications of the research
• A bibliography  of relevant sources

Dissertation proposals vary a lot in terms of length and structure, so make sure to follow any guidelines given to you by your institution, and check with your supervisor when you’re unsure.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Step 1: coming up with an idea, step 2: presenting your idea in the introduction, step 3: exploring related research in the literature review, step 4: describing your methodology, step 5: outlining the potential implications of your research, step 6: creating a reference list or bibliography.

Before writing your proposal, it’s important to come up with a strong idea for your dissertation.

Find an area of your field that interests you and do some preliminary reading in that area. What are the key concerns of other researchers? What do they suggest as areas for further research, and what strikes you personally as an interesting gap in the field?

Once you have an idea, consider how to narrow it down and the best way to frame it. Don’t be too ambitious or too vague – a dissertation topic needs to be specific enough to be feasible. Move from a broad field of interest to a specific niche:

• Russian literature 19th century Russian literature The novels of Tolstoy and Dostoevsky
• Social media Mental health effects of social media Influence of social media on young adults suffering from anxiety

Prevent plagiarism, run a free check.

Like most academic texts, a dissertation proposal begins with an introduction . This is where you introduce the topic of your research, provide some background, and most importantly, present your aim , objectives and research question(s) .

Try to dive straight into your chosen topic: What’s at stake in your research? Why is it interesting? Don’t spend too long on generalisations or grand statements:

• Social media is the most important technological trend of the 21st century. It has changed the world and influences our lives every day.
• Psychologists generally agree that the ubiquity of social media in the lives of young adults today has a profound impact on their mental health. However, the exact nature of this impact needs further investigation.

Once your area of research is clear, you can present more background and context. What does the reader need to know to understand your proposed questions? What’s the current state of research on this topic, and what will your dissertation contribute to the field?

If you’re including a literature review, you don’t need to go into too much detail at this point, but give the reader a general sense of the debates that you’re intervening in.

This leads you into the most important part of the introduction: your aim, objectives and research question(s) . These should be clearly identifiable and stand out from the text – for example, you could present them using bullet points or bold font.

Make sure that your research questions are specific and workable – something you can reasonably answer within the scope of your dissertation. Avoid being too broad or having too many different questions. Remember that your goal in a dissertation proposal is to convince the reader that your research is valuable and feasible:

• Does social media harm mental health?
• What is the impact of daily social media use on 18– to 25–year–olds suffering from general anxiety disorder?

Now that your topic is clear, it’s time to explore existing research covering similar ideas. This is important because it shows you what is missing from other research in the field and ensures that you’re not asking a question someone else has already answered.

You’ve probably already done some preliminary reading, but now that your topic is more clearly defined, you need to thoroughly analyse and evaluate the most relevant sources in your literature review .

Here you should summarise the findings of other researchers and comment on gaps and problems in their studies. There may be a lot of research to cover, so make effective use of paraphrasing to write concisely:

• Smith and Prakash state that ‘our results indicate a 25% decrease in the incidence of mechanical failure after the new formula was applied’.
• Smith and Prakash’s formula reduced mechanical failures by 25%.

The point is to identify findings and theories that will influence your own research, but also to highlight gaps and limitations in previous research which your dissertation can address:

• Subsequent research has failed to replicate this result, however, suggesting a flaw in Smith and Prakash’s methods. It is likely that the failure resulted from…

Next, you’ll describe your proposed methodology : the specific things you hope to do, the structure of your research and the methods that you will use to gather and analyse data.

You should get quite specific in this section – you need to convince your supervisor that you’ve thought through your approach to the research and can realistically carry it out. This section will look quite different, and vary in length, depending on your field of study.

You may be engaged in more empirical research, focusing on data collection and discovering new information, or more theoretical research, attempting to develop a new conceptual model or add nuance to an existing one.

Dissertation research often involves both, but the content of your methodology section will vary according to how important each approach is to your dissertation.

Empirical research

Empirical research involves collecting new data and analysing it in order to answer your research questions. It can be quantitative (focused on numbers), qualitative (focused on words and meanings), or a combination of both.

With empirical research, it’s important to describe in detail how you plan to collect your data:

• Will you use surveys ? A lab experiment ? Interviews?
• What variables will you measure?
• How will you select a representative sample ?
• If other people will participate in your research, what measures will you take to ensure they are treated ethically?
• What tools (conceptual and physical) will you use, and why?

It’s appropriate to cite other research here. When you need to justify your choice of a particular research method or tool, for example, you can cite a text describing the advantages and appropriate usage of that method.

Don’t overdo this, though; you don’t need to reiterate the whole theoretical literature, just what’s relevant to the choices you have made.

Moreover, your research will necessarily involve analysing the data after you have collected it. Though you don’t know yet what the data will look like, it’s important to know what you’re looking for and indicate what methods (e.g. statistical tests , thematic analysis ) you will use.

Theoretical research

You can also do theoretical research that doesn’t involve original data collection. In this case, your methodology section will focus more on the theory you plan to work with in your dissertation: relevant conceptual models and the approach you intend to take.

For example, a literary analysis dissertation rarely involves collecting new data, but it’s still necessary to explain the theoretical approach that will be taken to the text(s) under discussion, as well as which parts of the text(s) you will focus on:

• This dissertation will utilise Foucault’s theory of panopticism to explore the theme of surveillance in Orwell’s 1984 and Kafka’s The Trial…

Here, you may refer to the same theorists you have already discussed in the literature review. In this case, the emphasis is placed on how you plan to use their contributions in your own research.

You’ll usually conclude your dissertation proposal with a section discussing what you expect your research to achieve.

You obviously can’t be too sure: you don’t know yet what your results and conclusions will be. Instead, you should describe the projected implications and contribution to knowledge of your dissertation.

First, consider the potential implications of your research. Will you:

• Develop or test a theory?
• Provide new information to governments or businesses?
• Challenge a commonly held belief?
• Suggest an improvement to a specific process?

Describe the intended result of your research and the theoretical or practical impact it will have:

Finally, it’s sensible to conclude by briefly restating the contribution to knowledge you hope to make: the specific question(s) you hope to answer and the gap the answer(s) will fill in existing knowledge:

Like any academic text, it’s important that your dissertation proposal effectively references all the sources you have used. You need to include a properly formatted reference list or bibliography at the end of your proposal.

Different institutions recommend different styles of referencing – commonly used styles include Harvard , Vancouver , APA , or MHRA . If your department does not have specific requirements, choose a style and apply it consistently.

A reference list includes only the sources that you cited in your proposal. A bibliography is slightly different: it can include every source you consulted in preparing the proposal, even if you didn’t mention it in the text. In the case of a dissertation proposal, a bibliography may also list relevant sources that you haven’t yet read, but that you intend to use during the research itself.

Check with your supervisor what type of bibliography or reference list you should include.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Caulfield, J. (2022, November 11). How to Write a Dissertation Proposal | A Step-by-Step Guide. Scribbr. Retrieved 3 September 2024, from https://www.scribbr.co.uk/thesis-dissertation/proposal/

Is this article helpful?

Jack Caulfield

Other students also liked, what is a dissertation | 5 essential questions to get started, what is a literature review | guide, template, & examples, what is a research methodology | steps & tips.

• Faculty of Social Sciences
• School of Law
• Research degrees

Writing a research proposal

As part of the process of applying for a research degree, you will need to prepare an outline of your proposed research. 

Please see our guidance on what to include below, including word count:

*Word count excludes footnotes. 

• Browse by Year
• Browse by Warwick Author
• Browse by Department
• Browse by Funders
• Browse by Theses by Department
• Advanced Search

Law dissertations : a step-by-step guide

Lammasniemi, Laura (2021) Law dissertations : a step-by-step guide. London: Routledge. ISBN 9780367568771

This is the latest version of this item.

Research output not available from this repository.

Request-a-Copy directly from author or use local Library Get it For Me service.

Request Changes to record. --> Request Changes to record.

Law Dissertations: A Step-by-Step Guide provides you with all the guidance and information you need to complete and succeed in your LLB, LLM or law-related dissertation. Written in a simple, clear format and with plenty of tools to help you to put the theory into practice, Laura Lammasniemi will show you how to make writing your law dissertation easy, without compromising intellectual rigour.

As well as explaining the process of research and outlining the various legal methodologies, the book also provides practical, step-by-step guidance on how to formulate a proposal, research plan, and literature review. Unlike other law research skills books, it includes a section on empirical research methodology and ethics for the benefit of students who are studying for a law-related degree.

Packed full of exercises, worked examples and tools for self-evaluation, this book is sure to become your essential guide, supporting you on every step of your journey in writing your law dissertation.

Export / Share Citation

Request changes or add full text files to a record --> Request changes or add full text files to a record

Available Versions of this Item

• Law dissertations : a step-by-step guide. (deposited 17 Dec 2021 14:51) [Currently Displayed]

Repository staff actions (login required)

Writing for LL.M. Students

At HLS, students have excellent opportunities for writing in many different areas of the law, in a variety of different formats, and under the supervision of, or in collaboration with, many HLS faculty members. Faculty encourage and actively support student writing, working closely with students on papers written for academic credit, journal articles, and books.

The  Written Work Requirement  is a degree requirement for the LL.M. degree. Students interested in doing  additional writing  beyond the requirement may choose to write optional papers for writing credit.

In addition to writing for academic credit, HLS students may collaborate with faculty members, often serving as co-authors for books and law journal articles. Students also work closely with faculty to publish articles of their own. Students may also publish papers through the online student paper series sponsored by the HLS Library.

Many of our students are involved in reviewing, selecting, and editing faculty work for publication in Harvard’s  sixteen student-run law journals . Journal participation is a central part of the intellectual life of many HLS students; over 800 students work on journals each year.

For questions on LL.M. Writing Opportunities at HLS, please contact the  Graduate Program .

More Information about Written Work for LL.M. Students

• LL.M. Written Work Requirement
• Additional Written Work Opportunities
• Written Work Registration/Submission Deadlines
• Written Work Registration Forms
• Written Work Supervisors
• Winter Writing Program FAQ
• Upper-level J.D. and LL.M. Writing Groups

Modal Gallery

Gallery block modal gallery.

Capturing the unobservable in AI development: proposal to account for AI developer practices with ethnographic audit trails (EATs)

• Original Research
• Open access
• Published: 02 September 2024

Cite this article

You have full access to this open access article

• Yung-Hsuan Wu   ORCID: orcid.org/0009-0005-9723-5838 1  

The prevalence of artificial intelligence (AI) tools has inspired social studies researchers, ethicists, and policymakers to seriously examine AI’s sociopolitical and ethical impacts. AI ethics literature provides guidance on which ethical principles to implement via AI governance; AI auditing literature, especially ethics-based auditing (EBA), suggests methods to verify if such principles are respected in AI model development and deployment. As much as EBA methods are abundant, I argue that most currently take a top-down and post-hoc approach to AI model development: Existing EBA methods mostly assume a preset of high-level, abstract principles that can be applied universally across contexts; meanwhile, current EBA is only conducted after the development or deployment of AI models. Taken together, these methods do not sufficiently capture the very developmental practices surrounding the constitution of AI models on a day-to-day basis. What goes on in an AI development space and the very developers whose hands write codes, assemble datasets, and design model architectures remain unobserved and, therefore, uncontested. I attempt to address this lack of documentation on AI developers’ day-to-day practices by conducting an ethnographic “AI lab study” (termed by Florian Jaton), demonstrating just how much context and empirical data can be excavated to support a whole-picture evaluation of AI models’ sociopolitical and ethical impacts. I then propose a new method to be added to the arsenal of EBA: Ethnographic audit trails (EATs), which take a bottom-up and in-progress approach to AI model development, capturing the previously unobservable developer practices.

Explore related subjects

• Artificial Intelligence
• Medical Ethics

Avoid common mistakes on your manuscript.

1 Introduction

“If you want to understand the big issues, you need to understand the everyday practices that constitute them.” [ 1 ].

Artificial intelligence (AI) models’ Footnote 1 sociopolitical impacts have been widely recognized, especially given their gradual integration into various domains like healthcare, legal systems, insurance industries, etc. [ 2 , 3 , 4 , 5 , 6 ]. There have long been alarms sounding about the sociopolitical problems and ethical consequences these AI models engendered: Ivana Bartoletti challenged the perceived neutrality of data and exposed how deep-running real-world biases become modeled into machines in An Artificial Revolution [ 7 ]. Kate Crawford echoes the same sentiment, demonstrating in her Atlas of AI how the power dynamics and socioeconomic forces underlie the labor, data, classification systems, and outputs that lead to the creation of AI [ 8 ]. The field of AI ethics examines the sociopolitical and ethical issues that arise from AI models making significant decisions about humans and, therefore, advocate vehemently for a normatively grounded governance; an ideal form of AI governance should be based on a particular set of normative or ethical principles [ 9 , 10 , 11 , 12 ]. However, the promotion of ethical principles themselves is insufficient to ensure that they are respected throughout the development of AI models.

The rapidly growing multidisciplinary field of ethics-based auditing (EBA) is precisely dedicated to evaluating and verifying if ethical principles are implemented throughout AI development. EBA provides “a structured process whereby an entity’s present or past behavior is assessed for consistency with relevant principles or norms” [ 13 – 14 ]; it does so by translating abstract principles into “verifiable claims,” which are “statements for which evidence and arguments can be brought to bear on the likelihood of those claims being true.” [ 13 ]. EBA holds the promise of promoting procedural regularity, institutional trust, and transparency [ 15 , 16 , 17 ]. Not only has EBA garnered significant academic attention [ 15 ], but it has also piqued the interest of policymakers [ 18 ] and professional services firms [ 19 , 20 , 21 ]; a nascent yet headline-grabbing AI auditing industry is also emerging [ 22 , 23 , 24 , 25 ].

In this article, I argue that EBA’s existing methods rely on applying highly abstract ethical principles to specific cases in a top-down and post-hoc approach, which does not always allow the complete evaluation of the sociotechnical assemblage of AI models (Sect. 2). In fact, current EBA methods tend to focus more on the AI models themselves as if they are distinct from the developers’ practices and design processes that brought them into being in the first place; existing EBA methods then further hide the developers behind the curtain of mechanical objectivity and shielding them from scrutiny (Sect. 3). Using an ethnography framed as an “AI laboratory (lab) study,” whose methods are explained in Sect. 4, I expose how the creation of AI models is a value-laden and sociopolitical process (Sect. 5). I then propose a bottom-up and in-progress approach termed an ethnographic audit trail (EAT) to reveal the ethical weights stemming from developer practices and design processes (Sect. 6). In Sect. 7, I conclude with two identified challenges for keeping EATs and offer suggestions for future research to substantiate the proposed method.

2 Ethics-based auditing (EBA)

2.1 current methods.

The litany of abstract principles alone—whether grounded in ethics, human-centeredness, or other normative systems—provides neither sufficient guidance for AI developers to implement them nor concrete evaluation frameworks that could be used to verify AI developers’ self-proclaiming ethical practices. EBA comes into help: A variety of EBA procedures have been developed. The methods of translating abstract principles into verifiable claims are roughly divided along the lines of quantitative and qualitative procedures. The quantitative method relies on metricizing abstract values , constructing mathematical definitions of principles to describe the characteristics of training datasets and model performance [ 15 ]. The emerging field of Fair ML, for example, contributes a proliferating pool of tools that calculate fairness and measure algorithmic biases [ 26 – 27 ]. Footnote 2 Other principles like transparency and accountability are similarly quantified [ 15 ].

The qualitative method relies on operationalizing abstract values into verifiable statements . Consider AstraZeneca’s audit case [ 28 ]: The biopharmaceutical company hired an external auditor to examine both the high-level organizational structures and in-depth processes of its specific AI projects against its own published set of ethical principles [ 29 ]; The auditor took each principle and operationalized it into subsidiary and verifiable statements. For example, the principle of “Fair” was turned into two statements: “We endeavour to use robust, inclusive datasets in our Data [and] AI systems;” “We treat people and communities fairly and equitably in the design, process, and outcome distribution of our AI systems.” [ 28 ]. The auditor then collected data from interviews and company documentation to corroborate the boiled-down claims.

2.2 Applying a critical lens on ethics-based auditing

As much as EBA has gained traction as a way of assessing the ethical implications of AI models, I argue that the existing methods of EBA are insufficient to capture all the ethical implications that might arise from the specific contexts in which the AI models are born and function. Quantitatively metricizing and qualitatively operationalizing abstract values take a top-down approach to evaluate the ethical implications of AI models, assuming a preset of abstract ethical values that are universally relevant and applicable to different instances of AI models across contexts. For example, one significant draw of metricizing ethical values is that by rendering the latter mathematical, one can apply the metrics across the board as if they are but a technical standard neutral from humans’ subjective interpretations. This approach has been criticized for its “principlism” and “technical focus,” as it directly applies abstract concepts across models built for a range of complex contexts without considering the particularities and contingencies of reality [ 16 , 30 – 31 ]. On the other hand, qualitative operationalization of abstract values takes a similar top-down position; just as in AstraZeneca’s case, the auditor must start with a decided preset of values to operationalize [ 28 ]. This may lead the auditor to leave out information not readily understood in the conceptual framework defined by the preset and fail to capture new and emerging ethical risks.

Moreover, I argue that current EBA methods miss an essential piece of the puzzle—the developmental process of the AI model and AI developers’ specific practices that make the models capable of leaving any ethical impacts on the world in the first place. The current approaches in EBA are post-hoc in that ethical evaluations only occur after the model has been built, not before or during. Ethical metrics are often used to test model performance ; in other words, they are only used to evaluate the model’s capability to carry out ethical effects but not how they become capable. Reading between the lines of AI companies and researchers’ documentation on the often metric-based performance tests, they mainly concentrate on explaining the capacity of their models instead of detailing how the latter acquired such capabilities [ 32 , 33 , 34 ].

On the other hand, the qualitative operationalization of abstract values is devised for audits that look beyond the technical and examine the influences of governance structures, managerial decisionmaking, and documentation of design choices [ 35 ]. While holding the promise to capture the broader sociopolitical context of an AI model, this type of audit is often carried out via interviews and official documentation after the model has been developed and even deployed. This means that the auditor also misses out on many contextual details in which ethical risks arise, such as what competing alternatives existed for a specific ethically contested design choice and what factors determined the final decision.

Taken together, the current top-down and post-hoc approach in EBA inadvertently masked the critical process by which an AI model is developed, making the developers’ specific practices invisible. This is potentially an effect of the mechanical objectivity commonly associated with computer science and machine learning (ML) fields: The act of day-to-day coding and modeling is perceived to be purely based on practical functionality instead of sociopolitical considerations; hence, they do not need to be examined [ 36 – 37 ].

This mechanical objectivity of developers’ practices can lead to unintended, negative consequences. Ugwudike showed that the theoretical foundations of developers underpin their design logic, which then affects how predictive policing algorithms operate and generate real-world effects [ 38 ]. Marino also demonstrated that codes are more than functional instructions for machines but also personal expressions of the programmers [ 36 ]. From conceptualizing a real-world problem to be solved, coding instructions into scripts, assembling datasets that become ground truths to the machines, designing a model architecture, and devising reward functions to selecting performance metrics, AI developers’ implicit values and respective worldviews are embedded in the model every step of the way. AI models should not be considered immune from the sociopolitical influences of the larger world introduced by the hands of those who build them [ 39 ].

3 Discovering the new observable through ethnographic methods

There is a need to account for the new observable—the developer’s practices that mix in their values and worldviews into AI models—to reveal developers’ accountability by peeling away the façade of mechanical objectivity and to fully capture the root causes of ethical risks emanating from AI models. However, developers’ theoretical foundations and implicit worldviews are often not readily quantifiable and describable by metrics [ 38 ]. Qualitative methods like conducting interviews and reviewing developers’ documentation post-hoc are also insufficient; science and technology studies (STS) literature noticed that scientific reports and documentation tend to include only purified accounts of the decisions taken and provide step-by-step maxims of conduct in research or experimenting activities that discard and hide the scaffolding utilized to arrive at the scientific facts [ 40 ].

How can we capture this new observable? As informed by the STS literature, a non-top-down, non-post-hoc approach that allows us to examine developers’ practices and unearth values embedded in AI models might be ethnographies and, more precisely, “AI laboratory (lab) studies,” as proposed by Florian Jaton [ 40 ].

The justifications for an ethnographic method start with the theoretical reconceptualization of AI models as a sociotechnical assemblage: As Seaver observed, “algorithmic systems are not standalone little boxes, but massive, networked ones with hundreds of hands reaching into them, tweaking and tuning, swapping out parts and experimenting with new arrangements.” [ 41 ]. AI models “must be understood as composites of nonhuman (i.e., technological) actors woven together with human actors, such as data-creators, maintainers, and operators into complex sociotechnical assemblages.” [ 42 ]. Seen in this way, AI models are actively enacted by the practices of a myriad of actors that act on both technical and non-technical concerns [ 43 ]. The “intersection of dozens of…social and material practices” [ 44 ] that created AI models cannot be divorced from the broader contexts; an AI model must be understood as “relational, contingent, contextual in nature” instead of “technical, objective, impartial.” [ 37 , 45 ].

To unpack this complete sociotechnical assemblage, Kitchin proposed a combination of interviews, ethnographies, and document analyses, accounting for the “infrastructure/hardware, code platforms, data and interfaces” that are framed and conditioned by “forms of knowledge, legalities, governmentalities, institutions, marketplaces, finance and so on.” [ 37 ]. Consequentially, the venues for observation expand beyond the interview room where the managers and developers are arbitrarily taken outside of the scenarios where they practice and make decisions; instead, an observer must enter the exact places where AI models are under development and investigate the AI lab, the C-suite boardrooms, the cross-department meetings, developers’ desks, coding scripts, datasets, application interfaces, data contractors, front-end engineers, company competitors, market forces, and users.

By documenting the developers as they work and interact with technical and non-technical components, an ethnography is fitting to witness the “everyday practices that constitute [the algorithms] and keep them working and changing.” [ 43 ]. Jaton further considered a traditional analytical genre within STS called “laboratory studies.” [ 40 , 46 – 47 ]. Instead of starting from established scientific facts, lab studies concentrate on the “mundane actions and work practices to document and make visible how scientific facts were progressively assembled.” [ 40 ].

By definition, an AI lab study takes a bottom-up and in-progress approach to examining the development of AI models. It doesn’t start with a priori assumptions about what goes on in an AI lab and which set of values the ethnographer must pay attention to when documenting and problematizing practices, giving the ethnographer the total flexibility to note down any details. Moreover, as opposed to retrospectively examining a developed and deployed model, a lab study documents activities that occur in progress as “a set of intertwining courses of actions [which are accountable chronological sequences of gestures, looks, speeches, movements, and interactions among humans and nonhumans] sharing common finalities [such as ending up as a mathematical model, code, algorithm, or program].” [ 40 ].

In this paper, I apply AI lab study methods to a case to illustrate just how ethnographic methods can help capture previously unobservable developer practices. Especially bringing to the forefront the social practices developers engage in that influence the material practices, I will peel away the façade of mechanical objectivity of developer practices, showcase how developers encode subjective worldviews and social relations into AI models, and finally demonstrate why ethnographic methods to capture the new observable are critical in EBA and other ethical evaluations of AI models.

4 Methodology

There is no unified modus operandi of laboratory studies; there exist various viewpoints and approaches in the most renowned works [ 46 , 48 , 49 , 50 ]. Nevertheless, a few threads run through most studies. Taking a constructionist approach, lab studies examine scientific activities via direct participant observation that generates detailed, thick descriptions; ethnographers then use discourse analyses to make sense of the themes and related components underlying the dense qualitative data [ 47 ]. During a lab study, an ethnographer documents the “technical activities of science within the wider context of equipment and symbolic practices,” which treats the former as cultural activities [ 47 ]. In the same vein, technical objects are not to be considered “technically manufactured in laboratories” but “symbolically and politically construed.” [ 47 ].

The practical steps of my lab study are straightforward: I located an AI lab where I could gain sufficient access to both the AI developers and the technical artifacts they work on; took notes during a variety of courses of action within the lab; followed specific processes that were parts of bigger projects; partook in meetings; conducted interviews to clarify facts; and analyzed findings.

The single-case study occurred in an AI lab within a Swiss-Maltese non-governmental organization (hereinafter “the Foundation”). The Foundation conducts capacity development activities supporting small and developing states in diplomacy, particularly in internet governance and digital policy. Apart from conducting research on policy processes and training, the Foundation also develops in-house technological products to test how various digital and, particularly, AI technologies could help diplomats’ day-to-day work. I zoomed in on one of the Foundation’s AI projects, which involves building an AI reporting system that generates just-in-time reports from international conferences and events. Footnote 3

The data collection ran from October to December 2023. There were several sites for observation, primarily meetings with a different mix of people, one-on-one chats, semi-structured interviews, and self-explorations. The inquiry was hybrid in that I conducted in-person observation via two field trips to Belgrade, Serbia, where the AI lab is, and partook in online processes like team and brainstorming meetings. I collected audio and video recordings, Footnote 4 observation notes, Footnote 5 drawn illustrations, and text documents. I also accessed code snippets, OpenAI Playground, the Foundation’s application programming interface (API) for external contractors, and the Foundation’s AI applications for internal and external use. Footnote 6 I held semi-structured interviews and one-on-one chats to clarify notes and verify interpretations.

5 Case study

One of the main characters in this case is the AI reporting system (Fig.  1 ). Under the hood are multiple models, databases, and interfaces, all serving different purposes: The AI reporting system takes an audio-visual recording of a session, transcribes all speakers’ speeches via a transcription model (TM), and then generates summaries of various formats and knowledge graphs based on the transcripts via several summarization models (SMs) and a knowledge-graph generating model (KGGM). Several iterations of the AI reporting system feature an advanced function: an AI chatbot. Users can query a chatbot via an interactive interface, asking questions about the given conference in natural languages; the chatbot generates responses that summarize session transcripts, saving users the trouble of sitting through multiple sessions. The chatbot is a retrieval-augmented generation (RAG) LLM application. First, session transcripts go through a vector-embedding model (VEM) and are stored in a vector database. Second, when the user queries the chatbot, the latter doesn’t just rely on the pre-trained data of the underlying LLM; instead, it retrieves vectorized transcripts as the context that are relevant to the user’s query via a retrieval model (RM). Finally, the user’s initial query and the context are concatenated and sent to the chatbot’s response-generating model (RGM)—its underlying LLM. This way, the chatbot answers questions using specific knowledge of the given conference.

Foundation’s AI reporting system

In this case study, it is impossible to cover all facets of the AI reporting system, which has been in development for over two years at the time of writing. Instead, I select three vignettes to substantiate my claims that the developmental process and developer practices of AI models are neither mechanically objective nor devoid of sociopolitical influences, and that the ethical impacts stemming from them can only be captured via ethnographic inquiries. From the architecture to the intended capabilities of the AI model, the personal worldviews and value systems of developers and non-developers alike are embedded in this AI reporting system via developers’ social and material practices.

5.1 Vignette 1: the director’s problem

The first vignette attempts to show that the very conceptualization of an AI model is based on the subjective imaginations of whoever is behind it. This vignette starts from the beginning, even before the technical artifacts were woven into a complex system: the Foundation’s Executive Director had long held a vision for an intelligent system well before the AI lab developers began assembling one.

Back in 1992, the Director wrote his master’s thesis on developing a rule-based AI system to assign legal responsibilities during international environmental accidents. In codifying international laws in an AI model, he became absorbed in the challenges different epistemes bring about and various methods to clarify fuzzy logic in a rule-based system. His interest in solving those challenges culminated in his vision of creating a knowledge management system that assists the human thinking process. He argued in Knowledge and Diplomacy that knowledge management could improve efficiency in a diplomat’s work by gaining access to information, introducing a workflow that passes down information, automating routine activities during this workflow, and eventually retaining knowledge generated throughout this process [ 51 ]. At that time, he had a vision of this knowledge management system but not the means to build it.

Fast-forward to 2020, the Director had long established the Foundation, whose work areas included reporting from international conferences on digital policymaking and diplomacy. For many years, the Foundation employed human reporters to write just-in-time session summaries; however, the Director noticed an opportunity for creating a knowledge management system that could efficiently process a massive amount of information generated from such events and turn it into knowledge. Under his direction, the Foundation’s AI researchers began experimenting with using AI models to summarize events and streamline the reporting process.

Recall that the “theoretical framework or the creators’ interpretation of the task, problem, or issue the system is designed to address” will “inform key dimensions such as model architecture, data selection and processing, as well as the outputs.” [ 38 ]. In the Director’s envisioned knowledge management system, the technology product must not be a mere standalone tool but a part of a workflow that processes information . Footnote 7 The AI lab’s task, as per instructions, was not to create a transcribing tool or a summarizing tool; the lab was instructed to create a system that simulated and automated the entire workflow of reporting : the system accesses information in the form of audio-visual recordings, passes such recordings to the TM that outputs session transcripts, and then passes such transcripts to several SMs that output session summaries in various formats like talking points or per-speaker summaries. The workflow then splits into multiple streams, as the summaries could be directly sent to the Foundation’s or partner organizations’ websites or passed down to the KGGM or the chatbot. The Director’s vision of a knowledge management system elevated interoperability among multiple components to be a key dimension in the design; the inputs and outputs of each underlying model must conform to the same format for the seamless operation of the overall system.

Second, this particular way of problematizing reporting— automation of procedures through workflow —implicitly requires that reporting activities be routinized . According to the Director, a knowledge management system can automate activities as long as the latter can be logically described [ 51 ]. Therefore, the AI lab must describe “what reporting is” in serialized steps. As shown in Sect. 5.2., the AI lab resorted to finding abstract characteristics of human conversations and attempting to design generalizable rules that machines could follow when summarizing sessions. The routinization of reporting activities limits reporting to only the general steps that can be serialized and automated; missing from these steps are some spontaneous activities a human reporter might’ve taken, including researching online for additional information, emailing panelists for clarifications, or consulting colleagues for expertise.

From the first vignette, I show that conceiving an AI model that performs anything is a personal endeavor with subjective interpretations. The Director’s vision of creating a knowledge management system dictated, on a higher level, the task that the AI lab needed to tackle and further defined what the resulting AI reporting system was and did. AI models are not merely machines that solve our problems; they are our imagination of the world and its problems; they are our respective worldviews, personalities, ambitions, and desires reformulated, reconfigured, and translated into codes.

5.2 Vignette 2: everyone else’s problems

If the conceptualization of an AI model is personal, then one must ask who else is involved in such conceptualization and how different worldviews interact in creating the resulting model. In the second vignette, I show that the resulting AI reporting system was further refined and negotiated by a network of actors and their respective worldviews.

Just like any AI lab in an institution or a company, the Foundation’s AI lab does not exist in a vacuum; instead, the AI researchers work very closely with other teams performing different organizational functions. The AI lab is based in the Foundation’s Belgrade office, sharing spaces with four other teams: The course team prepares various courses that the Foundation delivers to diplomats or higher-education students. The reporting team takes care of the daily monitoring and updating of digital news and global policy trends on the Foundation’s website; it is also the team that used to provide live coverage of major international events, such as the UN General Assembly (UNGA) and Internet Governance Forum (IGF). The creative lab designs social media campaigns and visuals for all published material. The tech team manages the technical infrastructure, from websites to internal tools and applications, that allows the Foundation to function.

While the AI lab focuses on the research and design (R&D) of AI models and applications, the problems to which their research is supposed to provide answers often require more than writing a few lines of code to solve a mathematical puzzle. Instead, the AI lab reaches out to different actors within and beyond the Foundation to resolve those problems.

Recall the problem statement set by the Director: He wanted to create a knowledge management system that processes information and knowledge as generated from international events. This was highly abstract and open to further interpretation. The first layer of interpretation already happened when the Director and the AI lab decided on a workflow-simulating AI reporting system consisting of various models, all doing simpler tasks while producing interoperable inputs and outputs. But the problem must be further boiled down.

To routinize reporting activities, the AI lab must describe them in generalizable rules and instructions for each AI model. For example, to build an SM, the researchers must determine what the model could pick up from session transcripts (i.e., what to summarize). The AI lab manager (the Manager) conceived of these sessions as consisting of various conversations among multiple speakers; the issue was then to understand what could happen in a conversation. The AI lab brought in a linguist, who broke down conversations into questions and answers ; the linguist taught the AI lab various question types one could pose in a conversation and ways to detect which type they were (rhetorical, open, etc.). The next step was to understand the answers; the AI lab consulted a debater who framed responses in terms of arguments, which were then understood as key points with corresponding supporting facts. Taking in these lessons, the AI lab instructed the reporting system to take each speaker’s paragraphs from a transcript, extract key points and supporting evidence, and present a session summary in this format.

This layer of interpretation operationalized the abstract problem statement for a specific use case. The knowledge management system that was supposed to simulate a workflow from accessing information to accruing knowledge became one that extracted key points and supporting facts from transcripts based on speeches delivered during a session . How this interpretation process happened is crucial. Implicitly, the definitions of a session and the act of summarization in reporting activities were shaped by two actors’ opinions: a session is understood in question-answer pairs, and to summarize is to detect the question and dissect the response into key points and facts. The people the AI lab consulted actively shaped the latter’s understanding of what information was valuable to access and the particular way to create knowledge from it.

There were still other layers to the interpretation of the problem statement: what counts as a good way to create knowledge ? In other words, what is a good summary ? In preparation for deploying the AI reporting system for IGF 2023, the AI lab conversed with their colleague, the lead reporter (the Reporter) from the reporting team to learn what she would need from the AI-generated reports. It turned out that, to her, the most valuable information would not be what was said this year but what was said this year that was different from all previous years . The Reporter had years of experience covering IGF events; she already possessed knowledge of past main discussion points. Knowing that the main messages of such events usually vary little from year to year, she found only novelties in ideas or arguments valuable to her; such information would help her write a final report that identifies emerging trends in digital policy discussions. The way that the AI reporting system should transform information into knowledge now includes comparing current knowledge to historical knowledge , and this became a part of the AI lab’s ongoing research, especially as the Foundation launched the IGF Knowledge project aiming to do just that after the end of IGF 2023.

Another person who affected the evaluation of good summaries was me. During my inquiry, I participated in the AI lab’s research activities, finding ways to allow the SM to recognize more contexts and inter-relations between speaker’s speeches in a transcript. I was conscious that how I approached the task was only informed and shaped by my experiences of attending sessions at a handful of conferences and events. I generalized my learnings about the usual flow of conversation in a moderator-led panelist discussion, drew a few flowcharts (Figs.  2 (left) and (right)), and then coded a script instructing an SM to extract information accordingly.

(left) and (right) Flowcharts of a panelist discussion

Whether my and the Reporter’s beliefs about what counts as a good summary were widely shared is beside the point; likewise, whether the linguist’s and debater’s anatomy of a conversation was accurate does not matter. What matters here is that the initial problem statement was transformed into something specific and operationalized according to a handful of actors’ comprehension of the problem, the different interpretations they adopt for what a session is, their personal experiences dictating which information is important, and so on and so forth. Then, the AI lab translated these expert insights into something they could work with, something that machines could work with. The comments of the linguist, the debater, and the Reporter deeply affected how the AI lab described reporting to the AI models and, therefore, affected what the AI reporting system did, does, or will do in the future.

The second vignette demonstrated the social nature of the supposed technical problem that the reporting system was tasked with solving. An AI model developed in a lab is never shielded from the broader social environment; the very developers who work on the model interact, exchange, and learn from other actors who all hold their worldviews and values and thereby mediate and embed all these varying worldviews and values into the AI model. An AI model’s capacity is shaped by a network of actors and the worldviews they respectively hold, and the social relations among the actors further determine how these worldviews merge and become infused into the AI model. The constitution of an AI model is as social and personal as can be.

5.3 Vignette 3: the developers’ problem

The previous vignettes showcase how developers engaged in social practices with a network of actors to interpret the problem the AI reporting system was supposed to resolve. In the third vignette, I zoom in on the AI developers themselves—what was their own interpretation of the given problem? In other words, apart from what everyone else wished the reporting system to do, what did the developers imagine about the system’s capability?

Recall the chatbot feature of the AI reporting system that was based on the RAG technique. In a sequence I called the “RAG experiment,” where the Manager and an AI lab member were building an RAG pipeline, I observed their interpretations of good system performance (i.e., what AI models should do) emerge as they hit a nail.

The lab researchers were well aware of the imperfections of the AI reporting system. In the organization, they had the clearest vision about what the reporting system could or could not do; their desire to perfect the system to their expectations drove their motivation for conducting further experiments in prompting and RAG.

The chatbot feature of the AI reporting system was based on the RAG technique, which retrieved information via semantic searches (Fig.  3 ). Take the UNGA 78 AI Chat as an example: when a user asked a question, the AI Chat generated answers based on country statements made during UNGA 78 and provided a clear source. Behind the scenes was a mathematical transformation of texts called vectorization. All the UNGA 78 session transcripts first went through the VEM, which split the transcripts into text chunks by a desired size, such as paragraphs. Based on how semantically similar these text chunks were, the VEM assigned a directional value that showed the distance among these text chunks in a high-dimensional space. This formed the vector database. When a user asked, “What did the US say about cybersecurity?” The RM would calculate the semantic similarity between this question and the vectorized text chunks; going through the vector database, the RM retrieved the text chunks closest—most semantically similar—to the question. In this example, the RM might find text chunks containing both “the US” and “cybersecurity.” Finally, the RGM would generate a response using both the returned chunks and the initial user query.

RAG chatbot

During my period of observation, the Manager already saw a problem with the RM and frequently brought up the following example to elucidate his felt urgency on the lack-of-context conundrum: Imagine a text chunk including a sentence like “Donald Trump says that there should be a wall on the southern border of the US.” The meaning of this sentence could not be understood unless one knows the broader context—such as if Donald Trump was incumbent when he made such a statement. If he was, then the sentence probably conveyed the official stance of the US at that time; otherwise, it would be Mr. Trump’s personal stance. The Manager argued that an RM model—such as the one used by the lab at that time—only conducted semantic searches and could not capture context beyond the text chunk; it might mistake official country stances, retrieve the wrong paragraphs for the chatbot, and lead the user to believe in wrongful answers.

Teaching the RM to recognize context became the Manager’s ambition. Starting in December 2023, the AI lab began a new round of research crunch where they experimented with various retrieval techniques, text transformation methods, different data types, and RAG pipeline evaluation frameworks. Given that this was ongoing research at the time of writing, I could not describe further their conclusions. Nevertheless, there are a few things to highlight.

First, the Manager’s concern was of great ethical importance, although he never framed it that way. In Tsamados et al.’s mapping of the ethics of algorithms, they proposed a few categories where ethical problems arose from epistemic factors [ 52 ]: The semantic-based RM might retrieve paragraphs that are not relevant to the user’s query because it is incapable of recognizing contexts; it might establish a wrongful connection between the search query and the vector database and commit “apophenia,” the problem of “inconclusive evidence” when one sees connections where there is none [ 52 ]. This is due to the inner logic of the RM and the vectorized nature of the database—both of which decontextualize and extract the original texts from the broader context that cannot be captured in the database. To address this issue, where the chatbot hallucinates due to inconclusive evidence or invents information, the AI lab implemented a source attribution functionality: the UNGA 78 AI Chat has a “source” button that shows the retrieved paragraphs on which the chatbot generates answers. This design choice enables the users to meaningfully evaluate the quality and accuracy of the chatbot’s response, thereby mitigating the potential harms of inconclusive evidence. Moreover, the source attribution functionality alleviated the problem of “inscrutable evidence”—where users could not understand how the model provided a given response—and improved the transparency of the model’s operation to an extent; this further allows users to use the model’s response with more trust [ 52 ]. In short, the Manager’s concern and activities taken by the AI lab in response had direct ethical implications.

Second, the Manager did not frame his concern in ethical terms but instead as a system performance problem. In fact, most of the issues that the Manager and other AI lab members raised during meetings were described as system performance problems; the AI reporting system would not be considered a good system if it performed the tasks poorly or not as intended (i.e., not being able to retrieve the most accurate information in the database). Critical scholars have long documented the tendency for tech communities to understand AI models only in “rational concerns” and explain their “efficiency” and “optimality” from technical perspectives [ 37 ]. Such traditions of adopting mechanically objective views are deeply rooted in computer science and AI development. After all, the education and training of programmers are fraught with scientific papers, reports, and textbooks that follow “step-by-step maxims of conduct” or provide only “purified accounts” of scientific results [ 40 ]; all of this encourages programmers to forego “other knowledge about algorithms– such as their applications, effects, and circulation.” [ 41 ].

Third, suppose the first and second points both hold; then as the Manager addressed his concern framed in system performance terms, he was actually addressing ethical concerns about the AI reporting system. This leads me to a provocative argument: one cannot meaningfully separate the technical and ethical concerns; in reality, every design decision about an AI model is both technical and ethical. During model development, AI developers naturally encounter situations where their considerations about a design choice raise ethical implications. These situations emerge as developers deliberate about their options, take an erroneous turn, calibrate their course, and eventually lead down a particular path while leaving other routes behind. In this sense, ethics is practiced as it is contended, intentional or not, during the developers’ daily operations.

Consolidating what I have shown so far in the three vignettes, I argue that ethnographic inquiries akin to AI lab studies can reveal one critical source of AI models’ ethical impacts on the world: Developer practices. In vignettes 1 and 2, I showed that the very conceptualization of an AI model and the decisionmaking process around its development are highly subjective endeavors; in deciding what the AI model will be built to do, a network of actors, including both developers and non-developers, negotiate and compromise about their desires, needs, preferences, values, and worldviews. Actors with varying levels of social status also enjoy different degrees of influence over the model, with the Director being able to set high-level objectives and architecture of the AI reporting system and the others only making suggestions about the specifics of underlying models. With these power differentials, the developers’ practices in consulting others to interpret what the AI reporting system is supposed to do are not only social but political. In vignette 3, I further demonstrated that each of the developers’ technical design decisions is actually ethical; the material practices they engage in can be examined as ethical. Coupled with the fact that developers’ decisionmaking process is often influenced by the social network of which they are a part, the personal, social, and even political nature of developer practices revolved around building an AI model becomes evident.

6 Proposal: ethnographic audit trails

If the case study successfully proves the presence of various sociopolitical forces behind the constitution of AI models, then policymakers and ethicists face a challenge. Current top-down and post-hoc methods of EBA cannot capture developer practices in such great detail to expose the sources of the ethical implications of AI models. A bottom-up and in-progress approach to observing the ethical practices of developers—understood as every social and material practice related to the constitution of AI models—must be introduced as a potential EBA method. Although the methodologies of AI lab studies seem sufficient for the task, they remain too flexible to be standardized in regulations without more solid frameworks.

Fortunately, I can borrow a category of practices already existent in many programming projects: keeping an audit trail. Not to be confused with the auditing practices described in Sect. 2., an audit trail is a log of all steps taken to develop a specific system [ 13 ]. It is commonly used in the design of safety-critical systems such as commercial aircraft and financial industries, where step-by-step records of all decisions taken and resulting outcomes are kept [ 13 , 53 ]. In programming projects, the concept of audit trails is embodied by version control tools like GitHub and GitLab, which allow programmers to establish the traceability of changes made to all individual documents [ 13 ].

There are also precedents of keeping audit trails in the AI industry. Meta AI kept a chronical log as it trained its Open Pre-Trained (OPT) model [ 54 ]; Microsoft included an audit trail for their Azure AI Health Bot [ 55 ]. Some professional services firms and AI auditing companies are also offering automated audit trail tools [ 20 , 56 ]. Scholars have proposed to develop and standardize the requirements for AI audit trails: AI developers must provide chronological documentary evidence of the development of AI systems, including its problem definition, intended purpose, and design specifications [ 13 , 35 ].

Complementary to the existing practices, I propose what can be called “ethnographic audit trails (EATs)” to be performed not by developers but by social studies researchers, ethicists, or anthropologists embedded in AI labs. The existing practice of AI audit trails, especially the logs automatically kept by software, focuses primarily on what changes were made to the technical artifacts like coding scripts or datasets instead of the wider contexts in which such changes happen —meaning, the sociopolitical forces described in my case study are left out. An ethnographer should keep a distinct audit trail that contextualizes the decisionmaking process and critically examines the choices taken in an AI lab in relation to the constitution of a model from a sociological and ethical point of view. A typical AI lab study captures more than just the technical artifacts but also the social environment in which such artifacts are brought into being; likewise, an EAT is essentially a lab study structured around the chronological development of AI models that enriches the content of a regular AI audit trail.

The benefits of EATs are at least three-fold: First and foremost, EATs supply the empirical data needed to complement EBA methods and enable a complete life-cycle evaluation of AI models. Since an ethnographer undertaking an EAT starts with a bottom-up position—not assuming that a particular preset of ethical values applies in a given case, they will naturally generate a rich amount of empirical data throughout their inquiry. Such information produces great contexts for understanding how exactly ethical principles are considered on the ground and how high-level ethical principles can be interpreted on a granular level. Furthermore, the empirical data of EATs can serve as evidence for post-hoc auditing activities or any other governance compliance mechanisms under EBA. In this view, the approach of EATs does not contradict but instead complements existing EBA methods.

Second, the rich amount of contextual data generated by EATs can update the interpretation and operationalization of abstract ethical principles or even add new principles. EATs can capture unknown ethical and sociopolitical risk scenarios during AI development. By documenting the entire developmental process of AI models, an ethnographer might be able to identify emerging risks that are not foreseen given the current lack of documentation of developer practices in AI labs, thereby revealing the need to develop a new set of ethical principles. Moreover, the rapid pace of AI development calls for frequent updates to existing principles, such as what they might mean and how they can be applied in new applications or frontier developments. As innovations occur in AI labs, EATs allow ethicists to follow up with the moving boundary of AI development.

Lastly, given the position of an ethnographer, EATs hold the potential to promote a culture of ethical deliberations. Ethnographers carrying out EATs are embedded in a lab setting where they join meetings, participate in tasks, and exchange personal views with the people in and around the AI lab; these ethnographers can become integrated into the specific workplace culture, approach AI model development with interpersonal perspectives, and bring in more value-sensitive thinking into conversations. Recall vignette 3 (Sect. 5.3.): If every technical decision is simultaneously ethical, and if AI developers are essentially exercising ethics in their mundane yet daily operations, then the challenge of ensuring ethical designs of AI models must be resolved within the AI lab—where AI models are designed and developed. However, this challenge may be more difficult if developers never understand their AI models as ethically non-neutral and their practices as unobjective and sociopolitically significant; even if they do, they might still lack the vocabulary to frame their concerns in ethical terms and address emerging risks. By having ethnographers participate directly in AI lab activities and even presenting their findings and reflections periodically to the very actors they observe, ethnographers are best suited to inculcate AI labs with a culture of ethical deliberations that encourages developers to always consider the potential sociopolitical consequences of their actions and make decisions under the guidance of ethical principles. The notion of fostering a culture of ethics in AI development has gained traction [ 57 ], and it certainly aligns with various schools of design methods. Notable schools like human-centered design (HCD) and value-sensitive design (VSD) aim to sensitize tech developers to human and social concerns beyond mechanical functionalities [ 58 , 59 , 60 ]; with EATs, developers are further prompted to reflect in this regard, which may increase their sensibility about particular HCD methods such as inclusive and participatory design practices where different user groups and conventionally excluded communities are invited into design and decisionmaking processes [ 58 , 61 , 62 , 63 ]. In other words, by promoting a culture of ethical discussions among developers, EATs may motivate other well-established ethical design methods.

The proposal of EATs does not overthrow the need for existing EBA methods; in reality, EATs can complement EBA and substantiate high-level abstract principles. It is also recognized that EATs may be more resource-intensive and less scalable in comparison to other EBA methods, such as metricizing abstract values to create easily applicable benchmarks across models. However, the value of EATs does not lie in their scalability or capability of offering immediately comparable insights across cases; instead, EATs are most valuable when applied to domains where AI model development is nascent, fast-moving, or high-staked, such as AI for health, environment, humanitarian aid, or legal systems.

7 Conclusions

The sociopolitical impacts of AI models are indisputable, and their growing applications in different domains give rise to a sense of urgency for us to observe, identify, and mitigate such impacts. Researchers, policymakers, and ethicists have called for using high-level abstract value-based principles to guide the development of AI models. EBA arises as a natural instrument with which we can examine whether abstract principles are respected during AI models; however, I argue that existing EBA methods take a top-down and post-hoc position vis-à-vis AI development, which are not sufficient in capturing the developer’s social and material practices that encode the former’s personal values and worldviews into AI models in the first place. To account for the very developmental process of AI models and capture how the sociopolitical forces around the model are embedded in it, I propose adopting the methods of EAT to generate empirical data of such process.

Moving forward, there are two challenges to be tackled: The first is to testify and further substantiate the methodologies of an EAT. The current paper uses three vignettes in a case study to necessitate a modified method of empirical observation; however, the EAT has yet to be tested in the AI development scene. There remain questions about its feasibility and adaptability in all sorts of cases. A computer-automated audit trail in programming projects can easily keep track of all minor changes down to each line of code; an ethnographer-kept audit trail may still not be able to account for the wider contexts in which each minor change is made. An ethnographer carrying out such a task must acquire sufficient experience working with AI developers, programming languages, and many other technical artifacts to gain an intuition about which technical changes and design choices are significant, for which expansive coverage is needed, and which others are less significant. In other words, how exactly an ethnographer can carry out EATs and identify foci so as to facilitate proper documentation remains to be tested.

The second is to gain access to AI labs. Given the proprietary and lucrative nature of AI models and the fast-paced development of the field, AI labs may not always welcome an outsider to participate in their daily operations. One of the reasons I could gain access to the Foundation’s AI lab was my minimal capability of coding, which allowed me to become a useful member of the lab—or an insider . This status granted me the right to interact with some technical artifacts typically untouchable by non-programmers and non-members. Moreover, judging by the extent to which I have access, I detected there to be a “skill-to-access” ratio: the more programming skills one possesses to participate in difficult lab tasks, the more trust and membership privileges one is given, and the more access to technical artifacts one can be granted. For an ethnographer to carry out an EAT and partake in AI labs’ daily operations, one might need to overcome thresholds according to the skill-to-access ratio. The closer one wishes to be to the source of AI models’ sociopolitical power, the more one must understand about AI technologies.

To conclude, this paper identifies a niche yet to be filled in the wider debate about how to account for AI models’ sociopolitical impacts. The bottom-up and in-progress position that an EAT offers may complement existing EBA methods and ensure ethical considerations throughout the life cycle of AI model development.

In this paper, the term “AI model” refers to an algorithmic model that a computer builds partially without human intervention after observing some data and recognizing patterns from such data. The term “AI system” is used to describe an overall system consisting of multiple AI models.

Prominent Fair ML toolboxes include FAIRVIS, Microsoft Fairlearn, Google People and AI Research (PAIR)’s What-If Tools, IBM AI Fairness 360, University of Chicago Aequitas, etc.

I refer to a specific occasion of international conferences and events as “events”; there are usually multiple “sessions” at an “event.”

Audio and video recordings were obtained from meetings and interviews with participants’ consent to record. They were further transcribed into text files for analysis.

I took additional notes either on my laptop or in my notebook during meetings and interviews on top of the recordings to mark my observations and reflections on the spot. I separated reflections, which I considered the act of digesting, interpreting, and re-presenting what was being said and done during a process, from observations, which I considered the act of faithfully documenting what occurred during said process.

I could not retain copies of the technical artifacts as appendices to this paper since most are part of ongoing research and could only remain internal.

From here onwards to the end of Sect. 5.1., the phrases in italics are those rephrased or borrowed from the Director’s book Knowledge and Diplomacy , mainly from pages 8–9 where he describes a knowledge management system.

Suchman, L., Gerst, D., Krämer, H.: ‘If you want to Understand the Big issues, you need to Understand the Everyday practices that constitute them.’ Lucy Suchman in Conversation with Dominik Gerst & Hannes Krämer. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research. (2019). https://doi.org/10.17169/fqs-20.2.3252

Panch, T., Mattie, H., Celi, L.A.: The ‘Inconvenient truth’ about AI in Healthcare. Npj Digit. Med. 2 (1), 1–3 (2019). https://doi.org/10.1038/s41746-019-0155-4

Article   Google Scholar  

Richardson, R., Schultz, J., Crawford, K.: L Rev. Online. 94 , 192–228 (2019). https://ssrn.com/abstract=3333423 Dirty Data, Bad Predictions: How Civil Rights Violations Impact Police Data, Predictive Policing Systems, and Justice. 94 N.Y.U.

Yong, E.: A Popular Algorithm Is No Better at Predicting Crimes Than Random People. The Atlantic (blog). (2018). https://www.theatlantic.com/technology/archive/2018/01/equivant-compas-algorithm/550646/ Accessed 20 June 2024

Dressel, J., Farid, H.: The Accuracy, Fairness, and limits of Predicting Recidivism. Sci. Adv. 4 (1) (2018). https://doi.org/10.1126/sciadv.aao5580

Ferrer, X., van Nuenen, T., Such, J.M., Coté, M., Criado, N.: Bias and discrimination in AI: A cross-disciplinary perspective. IEEE Technol. Soc. Mag. 40 (2), 72–80 (2021). https://doi.org/10.1109/MTS.2021.3056293

Bartoletti, I.: An Artificial Revolution: on Power, Politics and AI. Indigo, London (2020)

Google Scholar  

Crawford, K.: Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press, New Haven, Connecticut, US (2022)

Dixon, R.B.L.: A principled governance for emerging AI regimes: Lessons from China, the European Union, and the United States. AI Ethics. 3 (3), 793–810 (2023). https://doi.org/10.1007/s43681-022-00205-0

Mäntymäki, M., Minkkinen, M., Birkstedt, T., Viljanen, M.: Defining organizational AI Governance. AI Ethics. 2 (4), 603–609 (2022). https://doi.org/10.1007/s43681-022-00143-x

Radu, R.: Steering the governance of Artificial Intelligence: National Strategies in Perspective. Policy Soc. 40 (2), 178–193 (2021). https://doi.org/10.1080/14494035.2021.1929728

Stix, C.: Actionable principles for Artificial Intelligence Policy: Three pathways. Sci Eng. Ethics. 27 (1), 15 (2021). https://doi.org/10.1007/s11948-020-00277-3

Article   MathSciNet   Google Scholar  

Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., et al.: Toward trustworthy AI development: Mechanisms for supporting verifiable claims. arXiv. (2020). https://doi.org/10.48550/arXiv.2004.07213

Mökander, J., Morley, J., Taddeo, M., Floridi, L.: Ethics-based auditing of automated decision-making systems: Nature, Scope, and limitations. Sci Eng. Ethics. 27 (4), 44 (2021). https://doi.org/10.1007/s11948-021-00319-4

Mökander, J., Floridi, L.: Ethics-based auditing to develop trustworthy AI. Mind. Mach. 31 (2), 323–327 (2021). https://doi.org/10.1007/s11023-021-09557-8

Brown, S., Davidovic, J., Hasan, A.: The Algorithm audit: Scoring the algorithms that score us. Big Data Soc. 8 (1) (2021). https://doi.org/10.1177/2053951720983865

Ayling, J., Chapman, A.: Putting AI Ethics to work: Are the Tools fit for purpose? AI Ethics. 2 (3), 405–429 (2022). https://doi.org/10.1007/s43681-021-00084-x

Information Commissioner’s Office: Guidance on AI and Data Protection. ICO. (2023). https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection/ Accessed 20 June 2024

PricewaterhouseCoopers (PwC): Responsible AI Toolkit. PwC. (2024). https://www.pwc.com/gx/en/issues/data-and-analytics/artificial-intelligence/what-is-responsible-ai.html Accessed 7 April 2024

Ernst: & Young (EY).: Responsible AI. EY. https://www.ey.com/en_ch/ai/responsible-ai . Accessed 7 April 2024

Deloitte: Trustworthy Artificial Intelligence (AI) ™ . Deloitte United States. https://www2.deloitte.com/us/en/pages/deloitte-analytics/solutions/ethics-of-ai-framework.html . Accessed 7 April 2024

Holistic, A.I.: What Is AI Auditing? Holistic AI. (2022). https://www.holisticai.com/blog/ai-auditing Accessed 7 April 2024

Fiddler, A.I., Observability, A.I., Model Monitoring, M.L., LLMOps, and, Explainable, A.I.: April. Fiddler AI. (2024). https://www.fiddler.ai/ . Accessed 7

Arthur: Observability. Arthur. (2024). https://www.arthur.ai/solution/observability . Accessed 7

Parity Consulting: Parity Consulting. Parity Consulting. https://www.get-parity.com . Accessed 7 April 2024

Pessach, D., Shmueli, E.: A review on Fairness in Machine Learning. ACM Comput. Surveys. 55 (3), 1–44 (2022). https://doi.org/10.1145/3494672

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A Survey on Bias and Fairness in Machine Learning. arXiv. (2022). https://doi.org/10.48550/arXiv.1908.09635

Mökander, J., Floridi, L.: Operationalising AI governance through Ethics-based auditing: An industry case study. AI Ethics. 3 (2), 451–468 (2023). https://doi.org/10.1007/s43681-022-00171-7

AstraZeneca.: Advancing Data and Artificial Intelligence. AstraZeneca. (2020). https://www.astrazeneca.com/sustainability/ethics-and-transparency/data-and-ai-ethics.html Accessed 20 June 2024

John-Mathews, J.-M., Cardon, D., Balagué, C.: From reality to World. A critical perspective on AI Fairness. J. Bus. Ethics. 178 (4), 945–959 (2022). https://doi.org/10.1007/s10551-022-05055-8

Lee, M.S.A., Floridi, L., Singh, J.: Formalising Trade-Offs beyond Algorithmic Fairness: Lessons from ethical Philosophy and Welfare Economics. AI Ethics. 1 (4), 529–544 (2021). https://doi.org/10.1007/s43681-021-00067-y

Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D., Gebru, T.: Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency. 220–29 (2019). https://doi.org/10.1145/3287560.3287596

OpenAI: GPT-4V(Ision) System Card. (2023). https://openai.com/research/gpt-4v-system-card Accessed 20 June 2024

MetaAI.: System Cards, a New Resource for Understanding How AI Systems Work. (2023). https://ai.meta.com/blog/system-cards-a-new-resource-for-understanding-how-ai-systems-work/ . Accessed 22

Mökander, J., Schuett, J., Kirk, H.R., Floridi, L.: Auditing large Language models: A Three-Layered Approach. AI Ethics. (2023). https://doi.org/10.1007/s43681-023-00289-2

Marino, M.: Critical Code Studies. The MIT Press, Cambridge, MA, US (2020)

Book   Google Scholar  

Kitchin, R.: Thinking critically about and researching algorithms. Inform. Communication Soc. 20 (1), 14–29 (2017). https://doi.org/10.1080/1369118X.2016.1154087

Ugwudike, P.: AI audits for assessing design logics and building ethical systems: The case of predictive policing algorithms. AI Ethics. 2 (1), 199–208 (2022). https://doi.org/10.1007/s43681-021-00117-5

Raji, I.D., Smart, A., White, R.N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., Barnes, P.: Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, 33–44 (2020). https://doi.org/10.1145/3351095.3372873

Jaton, F.: The Constitution of Algorithms. The MIT Press, Cambridge, MA, US (2021)

Seaver, N.: Knowing algorithms. In: Vertesi, J., Ribes, D. (eds.) digitalSTS: A Field Guide for Science and Technology Studies, pp. 412–422. Princeton University Press, Princeton, New Jersey, US (2019)

Chapter   Google Scholar  

Diakopoulos, N.: Transparency. In: Dubber, M.D., Pasquale, F., Das, S. (eds.) The Oxford Handbook of Ethics of AI, pp. 197–213. Oxford University Press, Oxford, UK (2020)

Seaver, N.: Algorithms as culture: Some tactics for the ethnography of Algorithmic systems. Big Data Soc. 4 (2) (2017). https://doi.org/10.1177/2053951717738104

Montfort, N., Baudoin, P., Bell, J., Bogost, I., Douglass, J.: 10 PRINT CHR$(205.5 + RND(1));: GOTO 10. The MIT Press, Cambridge, MA, US (2012)

S Geiger, R.: Bots, Bespoke, Code and the Materiality of Software platforms. Inform. Communication Soc. 17 (3), 342–356 (2014). https://doi.org/10.1080/1369118X.2013.873069

Latour, B., Woolgar, S.: Laboratory Life: The Social Construction of Scientific Facts. Sage, Beverly Hills, LA, US (1979)

Knorr Cetina, K.D.: Laboratory studies: The Cultural Approach to the study of Science. In: Jasanoff, S. (ed.) Handbook of Science and Technology Studies, pp. 140–167. Sage, LA, US (1995)

Knorr Cetina, K.D.: The Manufacture of Knowledge: An Essay on the Constructivist and Contextual Nature of Science. Pergamon, Oxford (1981)

Lynch, M.: Art and Artifact in Laboratory Science: A Study of Shop Work and Shop Talk in a Research Laboratory. Routledge Kegan & Paul, Boston, US (1985)

Traweek, S.: Beamtimes and Lifetimes: The World of High Energy Physicists. Harvard University Press, Cambridge, MA, US (1988)

Kurbalija, J.: Knowledge Management and Diplomacy. In: Kurbalija, J. (ed.) Knowledge and Diplomacy, pp. 7–19. Academic Training Institute, Msida, Malta (2002)

Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., Floridi, L.: The Ethics of algorithms: Key problems and solutions. AI Soc. 37 (1), 215–230 (2022). https://doi.org/10.1007/s00146-021-01154-8

Investopedia: Audit Trail: Definition, How It Works, Types, and Example. Investopedia. (2024). https://www.investopedia.com/terms/a/audittrail.asp . Accessed May 26

Meta Research: Metaseq/Projects/OPT/Chronicles at Main. Facebookresearch/Metaseq. (2024). https://github.com/facebookresearch/metaseq/tree/main/projects/OPT/chronicles . Accessed May 26

Microsoft Learn: Audit Trails - Azure AI Health Bot. (2023). https://learn.microsoft.com/en-us/azure/health-bot/audit-trails Accessed 20 June 2024

Credo, A.I.: Credo AI Audit Trail. Glossary. (2024). https://www.credo.ai/glossary/credo-ai-audit-trail . Accessed May 26

Langlois, L., Dilhac, M.-A., Dratwa, J., Ménissier, T., Ganascia, J.-G., Weinstock, D., Bégin, L., Marchildon, A.: L’éthique au cœur de l’IA., Obvia: Quebec, Canada (2023). https://www.obvia.ca/ressources/lethique-au-coeur-de-lia

Auernhammer, J.: Human-Centered, A.I.: The role of human-centered Design Research in the development of AI. DRS Bienn. Conf. Ser. (2020). https://doi.org/10.21606/drs.2020.282

Sadek, M., Calvo, R.A., Mougenot, C.: Designing Value-Sensitive AI: A critical review and recommendations for Socio-Technical Design processes. AI Ethics. (2023). https://doi.org/10.1007/s43681-023-00373-7

Friedman, B., Hendry, D.G.: Value Sensitive Design: Shaping Technology with Moral Imagination. The MIT Press, Cambridge, MA, US (2019)

Bjerknes, G., Bratteteig, T.: User participation and democracy: A discussion of Scandinavian Research on System Development. Scandinavian J. Inform. Syst. 7 (1), 73–98 (1995). https://aisel.aisnet.org/sjis/vol7/iss1/1

Neuhauser, L., Kreps, G.L., Morrison, K., Athanasoulis, M., Kirienko, N., Van Brunt, D.: Using design science and ariicial intelligence to improve health communicaion: ChronologyMD case example. Paient Educaion Couns. 92 (2), 211–217 (2013). https://doi.org/10.1016/j.pec.2013.04.006

Abascal, J., Nicolle, C.: Moving towards inclusive design guidelines for socially and ethically aware HCI. Interacing Computers. 17 (5), 484–505 (2005). https://doi.org/10.1016/j.intcom.2005.03.002

Download references

Acknowledgements

The author wishes to thank Florian Jaton for helpful comments on earlier versions of this article. The author also wishes to thank Jérôme Duberry and Oana Ichim for their guidance during the master’s thesis research. Finally, the author wishes to extend gratitude towards the organization where the inquiry takes place.

Open access funding provided by Geneva Graduate Institute.

Author information

Authors and affiliations.

Graduate Institute of International and Development Studies, Geneva, Switzerland

Yung-Hsuan Wu

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Yung-Hsuan Wu .

Ethics declarations

Consent for publication.

Written informed consent for publication of the personal details and/or images and/or audio recordings and/or videos was obtained from the participant/parent/guardian/relative of the participant. A copy of the consent form is available for review by the Editor of this journal.

Conflict of interest

The author did not receive any form of financial support from any organization for the submitted work. The author was doing a paid internship with the organization featured in the case study while carrying out research for this submitted work. However, the research was academically independent, as it was conducted as part of the author’s thesis research at the Graduate Institute of International and Development Studies instead of in his capacity as an intern at the case study organization.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Wu, YH. Capturing the unobservable in AI development: proposal to account for AI developer practices with ethnographic audit trails (EATs). AI Ethics (2024). https://doi.org/10.1007/s43681-024-00535-1

Download citation

Received : 21 June 2024

Accepted : 20 July 2024

Published : 02 September 2024

DOI : https://doi.org/10.1007/s43681-024-00535-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Ethics-based auditing
• Principle-based AI governance
• Ethnography
• AI audit trails
• AI lab studies

Advertisement

• Find a journal
• Publish with us
• Track your research