Getting Started with Applied ML Research
Advice on getting started with applied ML research
So you are interested in applied machine learning (ML) research? Oftentimes, a lot of young aspiring machine learning researchers jump straight into reading papers and either get discouraged with the amount of work published on a particular topic or get too caught up reading a lot of papers with very little progress on generating new and exciting research ideas. To avoid these situations and ensuring a healthy start on your research journey, here are some of my tips on how to get started with applied ML research. These tips do not guarantee success but they give you a useful framework for enabling creativity and avoiding common pitfalls in research.
Identify a problem that matches your background and skills
For instance, if you have studied biology maybe it makes more sense to conduct health-related research. If you have a physics background, there is plenty of interesting work contributing to space exploration/discovery.
It also matters that you work on a topic you are passionate about. If you are passionate about law, there is plenty of opportunities to propose new legislation to improve artificial intelligence accountability. This is the great thing about machine learning, it’s everywhere!
Once you have an idea on the topic you are interested to investigate, follow the next few steps.
Conduct a bit of research
Don’t read papers yet, just do a small search on articles or summaries of machine learning related techniques that have been applied in the domain of interest. Every day there are plenty of news outlets and newsletters that publish quick summaries of how companies and research labs are applying machine learning to all sorts of problems, these can help you find what you are looking for.
One great way to get informed about the topic of interest is to read survey papers or books on the subject matter. These are okay to read at this point. Remember, you are just looking on the surface.
I am of the belief that if it hasn’t been done before, then you need to think carefully about whether you will have the time to conduct the research you are interested in. This is the reason why there are Ph.D. programs to host students that want to work on entirely new research ideas and want to spend a good amount of years doing the research. If you are not interested in a Ph.D., that’s fine. Just make sure you have a good idea of how much time you are willing to invest in the research. Your idea doesn’t have to be completely new but it should tackle a fundamental problem you have identified.
This is important as it is here that you start to understand and begin to think about the motivation and importance of the research. Read with a critical mind. Think about what could be done better. What hasn’t been done? Take notes. Begin to formulate a rough draft of your research questions.
Look for implementations and datasets
No paper reading, yet. Paper reading can derail your progress at this point. Instead, begin to look for implementations of projects that tackle very similar problems as yours. If you can’t find code for similar ideas you want to tackle here is where you make an assessment of how much time you may need to commit. In the extreme case, you have access to tools and starter code to get you going. In the worst case, you have nowhere to start. The point is not that you are looking to copy code. Here you are mostly looking for work that has been done already so that you don’t have to start from scratch. This is the reason why open source exists.
For instance, if you will be working on cell segmentation, there are plenty of implementation for either cell segmentation or related tasks involving segmentation tasks. This could be your starter code. You will need to modify it, but a lot of that initial work has been done and you will be working on top of it. There is no shame in that. In fact, most scientists, even the greatest minds of our generation, rely on the ideas of others.
If you do find code, make sure to test it and see if it’s still running. It may be filled with bugs which means there is a bit more time you need to commit here. But try to get it to work.
If you don’t find code, then at least have a plan of what kind of implementations you will be coding. I am sure there is plenty of basic examples out there that can also be reused. Make sure to always provide the proper credit.
Remember, a lot of the research will be about implementation and experimentation and so you will need to get your hands dirty with code unless you are working on a big team.
To find relevant code, search in places like GitHub, Papers with Code, or Made with ML. If you can’t find anything there, you can always ask on Reddit or Twitter. There is a very helpful community out there.
In this step, it is also important to identify datasets you will be working with initially. The great thing about looking for implementations is that they already provide you a hint on the type of dataset you will be working with. You can perform dataset searches on websites like Google Dataset Search or Kaggle. In the best case, you find something that you think will help. In the worst case, you don’t find anything relevant in which case you may need to manually collect your datasets which is a tedious and time-consuming task on its own. Here is where you think carefully about whether you have the necessary time to collect and manually label a new dataset. Consult with your research advisor on this topic. I will write about this topic later on in this blog. If you want to keep on reading about my research and engineering experiences subscribe below to receive upcoming articles directly in your inbox:
Write a draft of your proposal
After having an idea of the topic you want to focus on and the time you may need to commit to this project, write a short 2-pages draft of your proposal.
Your proposal should have at least an introduction, motivation for the research, proposed methodology and experiments (it doesn’t have to be all fleshed out yet), the timeline of the research, and the expected output.
After finishing that draft, don’t let it just sit there. Pass it by a colleague first, then refine it. Try to reach out to friends and let them review it as well. Refine it a bit more.
Talk to your research advisor and other researchers
Even though it is highly likely that you are working on this research on your own, you have a plethora of people and experts that can provide you guidance. Discuss your proposal with your research advisor or other experts in the field. There is this misconception that research is a lonely process, it doesn’t have to be that way.
Expect a lot of feedback mostly about what the proposal is lacking and what potential new ideas you can try or things you overlooked. Take this feedback with an open mind. Your ideas may be great but they could also be very marginal or not worth pursuing. However, there is always something good that comes out of this process. You know a lot more about what you are interested in and what you want to work on. If you need to go back to the drawing board do so. This is rarely the case but just be prepared and keep an open mind. If all goes well, use the feedback to keep improving your proposal and refine those research questions. Once you have finalized the research questions, move on to the actual research.
The real research begins
Once you have some starter code and datasets that you think will help you in your research, it’s now time to begin addressing those research questions. By this time, you have an idea more or less what are the problems worth addressing and are possible given the time constraints and data available.
Adopt code and begin to conduct preliminary experiments. Very basic experiments could include simple baselines using basic ML algorithms designed to address the type of problem you are working on. For instance, if you are working on text classification, maybe an implementation of a Naive Bayes classifier could be a good model to start with. You don’t have to jump straight into a state of the art deep learning model like RNN or Transformer, although those usually perform better on these tasks considering you have enough data. But just keep it simple at the beginning. Later on, you will have enough time to experiment with those fancy models. All you are interested in here is to have a proper framework for experimenting.
This is where things get really interesting and challenging. While you are working on code and experiments, at this point in time it wouldn’t hurt to look for relevant papers to see if others have conducted similar experiments. Use that knowledge to keep improving your experiments. For instance, maybe there is a feature engineering trick you would like to try or a new technique to preprocess data. In the process, make sure to save those references. When you are stuck with code, read a few more papers, and continue where you left off. Mix it a bit. Too much coding or too much paper reading is not a good recipe for success as a researcher. It’s just simply not sustainable.
Keep building on top of that knowledge and revise your research questions if needed. You avoided paper reading initially mostly to avoid being discouraged about others scooping your work and ideas. This tends to happen a lot but because you didn’t read papers in the beginning it could well be that your ideas offer a new take on the problem. In fact, if you chose this route I proposed here, there is a high chance that there is a bit of overlap and there is also a high chance there is something new and creative you are bringing to the table. I think it’s very important to avoid reading papers at the beginning as there is a high chance that you are influenced to work on similar things to what other researchers have done. That’s not interesting and it’s actually not a good practice. You want to work on a problem that’s important to you, not a problem that is popular in the community.
Get it out there
Once you have some results, get your work out there. Blog about it even if no one reads it. Tweet about it. There is a whole ML community out there waiting to hear about exciting new ideas. In fact, I also built a community (dair.ai) to help surface the work of people that don’t have a platform.
Part of being a researcher is also being able to communicate your work properly. There is no running away from it. By getting it out you are practicing for the final step in the research journey: either a publication, thesis defense, or pitching it to your manager for consideration as a product.
Excellent. Keep working on those ideas. If new ideas come along consider tackling them. These days a lot of students are quick to publish. Don’t be one of those. Take your time. Enjoy your research. Dig deep. Commit to a long-term relationship with your research. It pays off tremendously in the long run. Learn more interesting things about the topic. Challenge yourself to go further. Don’t be too stressed about publishing. That time will come.
The point is that you should spend some time refining your work before you consider publishing it. Getting it out there helps with feedback and other considerations.
Publish it!
After many iterations and feeling a bit more comfortable with the progress of your research, it is now time to consider submitting it to workshops or conferences. Although some universities or companies don’t force you to publish, I truly believe as a researcher you need to have a taste of the peer-review process. A rejection doesn’t say anything about you as a researcher. This is an opportunity to get others excited about reading and assessing your work. If you get a publication, great for you! This is the kind of progress you want to show as a researcher. You are headed in the right direction. The next step is to keep building on top of this research. Get others involved. Collaborate. You have a bit of experience now and that will take you a long way as a researcher. Be patient. Accept rejections. Keep learning and keep a healthy mind and body. But more importantly, keep an open mind.
Happy researching!
This article is a rough draft and will continuously be revised. I didn’t rigorously check for grammar correctness. If you have any questions or need additional clarification, please comment below. Don’t forget to subscribe for new upcoming articles related to ML research and engineering.