
Senior Software Engineer — AI Evaluation & Benchmarks
Alignerr · Posted 1 month ago
- India (Remote)
- Remote
- Contract
- 4+ yrs
- ₹8L - ₹10L
About the role
About The Role
What if the code you write could determine how smart the next generation of AI truly is? We're looking for experienced Software Engineers to design and build the coding benchmarks and data pipelines used to evaluate frontier AI models — the systems that decide whether an AI can actually reason, debug, and write production-quality software.
This is high-impact, technically demanding work at the intersection of software engineering and AI research. You'll work with large codebases, multiple programming languages, and scalable infrastructure to create evaluation systems that push the boundaries of what AI can do.
This is a fully remote contract role. If you thrive in fast-paced engineering environments and want your work to directly shape the trajectory of AI — this is the role.
- Organization: Alignerr
- Type: Hourly Contract
- Location: Remote
- Contract Length: 3 Months
- Commitment: Full-time availability preferred
What You'll Do
- Design and implement coding benchmarks used to evaluate frontier AI models across real-world programming tasks
- Build and maintain scalable data pipelines for AI evaluation workflows
- Analyze AI-generated code for correctness, reliability, and edge-case failures
- Create structured evaluation scenarios that rigorously test reasoning, debugging, and code quality
- Work with large code repositories and multi-language environments
- Collaborate on systems that improve how AI models understand and generate software
- Provide detailed technical feedback on model performance and failure patterns
- Contribute to the design of evaluation frameworks that set industry standards
Who You Are
- 4+ years of professional software engineering experience — this is non-negotiable
- Experience working at a high-growth tech company or top-tier software organization
- Expert proficiency in Python — you write clean, performant, well-tested Python code
- Hands-on experience with code repositories and working in large, complex codebases
- Proven experience designing and implementing LLM coding benchmarks and data pipelines
- Track record of working in high-performance engineering environments with large-scale products or platforms
- Strong command of version control systems (Git) and modern development workflows
- Bilingual or native English speaker with strong written communication skills
- Self-directed, technically rigorous, and comfortable operating with autonomy
What Makes a Perfect Match
Candidates with these additional qualifications have the highest chance of success:
- Senior or Lead-level engineering profiles with a history of technical ownership
- Bachelor's or Master's degree in Computer Science, Machine Learning, or a related field — or equivalent professional experience
- Proficiency in one or more additional languages: JavaScript, Go, C++, or other relevant languages
- Experience with CI/CD pipelines and writing robust unit tests (pytest, Mocha, JUnit)
- Background in security engineering or significant open-source contributions
- Familiarity with AI/ML evaluation methodologies or model benchmarking
Why Join Us
- Work on cutting-edge AI evaluation projects alongside world-class research teams
- Fully remote — work from anywhere with a reliable internet connection
- Your benchmarks directly influence how the most advanced AI systems in the world are measured and improved
- Freelance autonomy with meaningful, high-stakes engineering work
- Collaborate with a global community of elite engineers and researchers
- Potential for contract extension and ongoing engagement as new evaluation challenges emerge
Skills
Quark9 guide
How to land this Senior Software Engineer — AI Evaluation & Benchmarks role
Our notes for applicants, based on how data and AI hiring usually works and on 4 similar open jobs on Quark9.
Resume checklist for this job
Most companies screen resumes with an ATS before a person reads them. Make sure each skill you genuinely have appears in your resume in the same words the job uses.
Must have
- Python programming
- Version control systems (Git)
- AI model benchmarking and evaluation
- Lead each project with the business result (revenue, cost, time saved), then the method
- List the stack precisely: SQL dialect, Python libraries, BI tool, cloud platform
- Link a portfolio or GitHub with one or two well-documented projects
What interviewers expect: early to mid level
Expect questions about work you owned end to end. Show what you delivered, the tools you used daily and one or two results with numbers.
How data and AI interviews usually run
- A screening call on your projects and the size of data you have worked with
- A SQL round, usually joins, window functions and aggregations on a sample schema
- A case or modelling round: frame a business problem, choose metrics, explain trade-offs
- A take-home or live exercise in Python, then a final round with the hiring manager
Questions worth preparing
- Write a query for the second-highest salary in each department.
- How would you measure whether a new feature improved retention?
- Explain overfitting to a non-technical stakeholder and how you guard against it.
- Walk through a project end to end: the question, the data, the method and the impact.
Every company runs its own process, so treat this as a likely shape, not a promise. Practise a mock interview on Quark9 and get scored feedback before the real one.
The market for Software Engineer — roles right now
- Most openings are in Mumbai, Bangalore, Noida, and 1 of the 4 we looked at offers remote work.
From open jobs on Quark9 matching “Software Engineer —”, updated every few minutes.
Similar jobs
- Staff Software Engineer — Active Directory Internals & Directory ServicesCohesity · Bangalore Rural, Karnataka, India (On-site)
- QA Automation EngineerWSP in India · Noida, Uttar Pradesh, India (Hybrid)
- Systems software engineer — c++/rust, backendJobCrexa · Mumbai Metropolitan Region (Remote)
- Front-End Software EngineerDeltatre · Mumbai, Maharashtra, India (Hybrid)