Muhammad Roshaan · AI Safety & Systems Engineer

Rigorous evals.
Resilient systems.

Final-year CS at SSUET. First author of an accepted SLM-Agents paper at NeurIPS 2026, building Bayesian evaluation tooling and backend systems that hold under contention.

GitHub Résumé Email LinkedIn

2

accepted papers at NeurIPS 2026 workshops

4

AI safety and LLM evaluation contributions

3.80

CGPA · BS Computer Science, SSUET

6

certifications from Udemy, DataCamp, and Cisco

Python•Go•FastAPI•pytest•GitHub Actions•OpenTelemetry•vLLM•Qwen3•Gemini•PostgreSQL•Redis•Celery•Docker•SQLAlchemy•MongoDB•scikit-learn•XGBoost•Power BI•Python•Go•FastAPI•pytest•GitHub Actions•OpenTelemetry•vLLM•Qwen3•Gemini•PostgreSQL•Redis•Celery•Docker•SQLAlchemy•MongoDB•scikit-learn•XGBoost•Power BI•

01 · Research

Evaluations that measure what matters

Work on LLM tool calling, agent tracing, and safety measurement, plus reviewing for a NeurIPS 2026 workshop.

SLM-Agents @ NeurIPS 2026 · Paris
Accepted · poster

BitFracture

Compression Breaks Discipline: Quantization Increases Missed Tool Calls in Small Language Models, but Not Mid-Size Ones

First author · Jul 2026 - Oct 2026

A pre-registered audit of how 4-bit NF4 quantization changes the distribution of tool-calling errors in small models, instead of reporting a single accuracy number.

  • Qwen3-1.7B missed required tool calls nearly tripled under NF4 (6.0% to 17.3%, paired McNemar p=0.0001), and 13 of the 26 NF4 misses were correct under FP16.
  • Qwen3-4B showed a powered null, and a direct size x format interaction test confirmed the asymmetry (-11.3pp, 95% CI [-20.0, -3.3], p=0.006).
  • Method: controlled FP16 vs NF4 comparison, a 7-class error taxonomy, bootstrap 95% CIs, and a phase-gated pilot (n=50) followed by a powered run (n=150 per cell).
Python
bitsandbytes
BFCL
Qwen3
Kaggle
Paper and analysis
AIWILD @ NeurIPS 2026 · Sydney
Accepted · poster

CanaryGame

Shared-Memory Honeytokens in Multi-Agent LLM Systems: A Controlled Simulation

Co-author · Aug 2026 - Dec 2026

A controlled simulation testing whether shared-memory coalitions defeat honeytoken decoys in multi-agent LLM teams, run fully locally with no real credentials or network egress.

  • Brokered containment pinned post-trigger harm at exactly zero (95% CI [0, 0]) with the highest trap activation (0.80), across Qwen3-4B, Qwen3-8B, and Gemma3-4B.
  • Shared-memory coalitions pooled decoy fingerprints (contagion up to 16) and raised mean harm over baseline.
Python
vLLM
Qwen3
Gemma3
Agent Simulation
Verify-Agents and AIWILD @ NeurIPS 2026
Under review

Agent Watchtower

CausalTrace: Order-Invariant Causal Trace Verification

Primary author · Aug 2026

Collaborative research on CausalTrace, an order-invariant verification engine for concurrent multi-agent traces: a Go canonicalizer, a Python serialization sweep, and an LLM-judge study.

  • Built a Go canonical causal-DAG renderer with byte-stable checksums and a 160-cell sweep over 40 seeded traces across 4 fault conditions.
  • Deterministic verdicts were order-invariant (0% flips across topological serializations), while live LLM judges flipped on 55% to 75% of identical facts (cross-family kappa 0.08 to 0.29).
  • Ran 240 live judge calls across Gemini, DeepSeek, and Kimi, kept 127+ offline tests green, and merged the contribution upstream.
Go
Python
OpenTelemetry
Gemini
DeepSeek
Kimi
Repository
NeurIPS 2026 submission · SPAR AI mentee project
Under review

When the Judge Drifts

RLVR Safety Evaluation (title withheld during double-blind review)

Contributor · Aug 2026

An RLVR safety-evaluation study that separates changed behavior from changed measurement along a GRPO training trajectory (Tulu 3.1 8B, 12 pinned checkpoints, 6,912 structured responses).

  • The marginalized safety score never left its prespecified ±0.10 band, while measurement properties drifted (permutation invariance -0.146, answer-order range +0.188).
  • The study attributes the change to measurement drift rather than safety drift.
  • Built on a 24-source counterbalanced safety instrument (72 wordings x 6 orderings) with frozen claim gates and blinded AI reviewers.
Python
RLVR
GRPO
Bootstrap Statistics
Gemini

Peer review

Reviewer, Verify-Agents Workshop @ NeurIPS 2026

Nominated by the program committee and accepted to review for the workshop on reliable agent development.

Aug 2026

02 · Selected work

Systems, chosen to explain themselves

Case studies over links. Each project has a page walking through the problem, the approach, and what actually changed.

01 / Backend · Distributed Systems

SeatVault

Backend system for high-contention reservations with atomic inventory control, Redis distributed locking, PostgreSQL row locks, and idempotent payment-style workflows.

Python
FastAPI
PostgreSQL
Redis
Celery
Docker
02 / AI/ML · MLOps

Sentinel

Fraud-detection service combining real-time event processing, machine learning inference, explainability, and operational monitoring.

Python
FastAPI
scikit-learn
XGBoost
LightGBM
Redis
PostgreSQL
Hopsworks
03 / Full Stack · Offline-First

BooknScore

Offline-first tape-ball cricket scoring and tournament platform with player and team analytics, authentication, synchronization, and a companion AI service.

Flutter
Supabase
PostgreSQL
SQLite
Python
Gemini
LLM Agents
04 / AI Safety · FYP

SafetyShield

Registered final year project (team of 4, SSUET): a Bayesian-efficient AI-safety evaluation platform that detects risky model behavior with far fewer prompt evaluations, plus a guardrail add-on that reduces unsafe outputs.

Python
FastAPI
Bayesian Statistics
LLM Evaluation
Guardrails
React
05 / Data Engineering · ETL

Inventory ETL & Alerting Pipeline

Data pipeline that extracts inventory information, transforms and validates records, loads structured output, and generates alerts for operational conditions.

Python
FastAPI
PostgreSQL
SQLAlchemy
Celery
Redis
Docker

More on GitHub

Smaller builds and open-source work that did not need a full case study.

Tool-Calling Reliability Benchmark

Open source · LLM evaluation

Open-source contribution: merged a GitHub Actions CI workflow that runs the core pytest suite on every push and pull request, and fixed the evaluation summary's diagnostic-label counting with regression tests.

Python
pytest
GitHub Actions

FlashSale Concurrency Engine

Backend · Concurrency

High-contention flash-sale backend focused on inventory correctness, atomic reservation logic, and race-condition prevention.

Python
FastAPI
PostgreSQL
Redis
Celery
Docker

Geospatial Fleet Dispatch API

Backend · Serverless

Serverless real-time fleet dispatch ingesting driver coordinates and mapping nearby vehicles through geospatial database queries and a command-center interface.

FastAPI
MongoDB
Cloudflare Workers
Next.js
WebSockets

Dealer & Vehicle Inventory Module

Backend · Multi-tenant

Java Spring Boot modular monolith with tenant-aware data isolation, custom query filtering, and role-based access control.

Java
Spring Boot
Spring Data JPA
Maven

FastAPI Book Manager API

Backend · API

CRUD API for book management using FastAPI, MySQL, SQLAlchemy ORM, and Pydantic validation.

Python
FastAPI
MySQL
SQLAlchemy
Pydantic

Movie Ticket Booking System

Desktop · Database

Python GUI application for movie ticket booking with relational data management, authentication, and seat reservation logic.

Python
MySQL
GUI

Netflix Content Analysis

Data · Analytics

Data cleaning, exploration, and visualization project using the Netflix dataset.

Python
Jupyter Notebook
Pandas

Shopping Mall Sales & Customer Insights

Analytics · BI

Power BI dashboard analyzing sales and customer behavior through revenue trends, age-group spending, product categories, and payment methods.

Power BI
DAX

Social Sentiment Dashboard

Analytics · BI

Power BI dashboard visualizing sentiment patterns across social media platforms using hashtag-level data.

Power BI
DAX

03 · Capabilities

The stack I reach for

Languages, frameworks, and methods, grouped by where they earn their keep in my work.

01

Backend & APIs

Services, contracts, and background work.

Python
FastAPI
Java
Spring Boot
REST APIs
WebSockets
SQLAlchemy
Pydantic
02

Concurrency & Distributed Systems

Keeping state correct when requests race.

Distributed Locking
Race-Condition Prevention
Atomic Transactions
Idempotent Workflows
Asynchronous Processing
Celery
Redis
03

Databases & Cloud

Relational, document, and in-memory stores.

PostgreSQL
MySQL
SQLite
MongoDB
Supabase
SQL
Database Design
Data Synchronization
Docker
Cloudflare Workers
04

Data Engineering & Analytics

Pipelines, validation, and reporting.

Pandas
NumPy
Matplotlib
ETL Pipelines
Data Cleaning
Data Validation
Power BI
DAX
Jupyter
05

Machine Learning & MLOps

Models from feature to deployment.

scikit-learn
XGBoost
LightGBM
Feature Engineering
Model Evaluation
Hopsworks
Model Deployment
MLOps
06

Generative AI

LLM features and agent workflows.

Gemini
LLM Agents
Multimodal AI
Prompt Engineering
Model Deployment
07

AI Safety & LLM Evaluation

Measuring models honestly.

LLM Evaluation
Tool Calling
Fault Injection
Bayesian Statistics
Guardrails
AI Alignment Concepts
08

Systems Engineering

Designing for load and failure.

High-Concurrency System Design
Zero-Data-Loss Pipelines
Data Integrity
System Stability Under Load
Load Balancing
Horizontal & Vertical Scaling

04 · Experience

Where the work happens

Scientific benchmark evaluation in progress, applied data science completed, both remote.

AI/LLM Evaluator @ Scientific AI Evaluation

Sep 2026 - Present

Physics & Materials Science · Remote

  • Review Physics and Materials Science research papers to find problems that need real numerical or computational work, not formula substitution.
  • Design self-contained Jupyter Notebook benchmarks with a main problem, two sub-problems, reference solutions, and executable test suites.
  • Validate scientific correctness, numerical accuracy, domain boundaries, and reproducibility with appropriate floating-point tolerances.
  • Evaluate Gemini, GPT, and Claude across repeated runs, diagnosing failures and refining prompts, reference implementations, and test coverage.
Python
NumPy
Jupyter
Google Colab
LaTeX
Gemini
GPT
Claude

Data Science Intern @ 10Pearls Pakistan

Apr 2026 - Jun 2026

10Pearls Shine Internship Program · Remote

  • Built and evaluated machine learning workflows for practical data science tasks.
  • Applied data preparation, feature engineering, model evaluation, and experiment tracking concepts.
  • Worked with Python data and machine learning tooling in a structured internship environment.
Python
Pandas
NumPy
scikit-learn
XGBoost
LightGBM
Hopsworks
Jupyter

05 · Foundations

Education and certifications

The degree in progress, the courses behind it, and the credentials collected along the way.

Sir Syed University of Engineering & Technology

BS Computer ScienceKarachi, Pakistan
Batch 2023FRoll No 2023F-BCS-311Regular (Morning)

Sep 2023 - Jul 2027

Status

Final year · 7th semester in progress

CGPA

3.80 / 4.00

Progress

102 credit hours earned

Semester GPAs

3.70 Fall 20233.71 Spring 20243.90 Fall 20243.92 Spring 20253.81 Fall 20253.78 Spring 2026

Selected coursework

Machine Learning
Artificial Intelligence
Data Structures and Algorithms
Database Systems
Operating Systems
Software Engineering
Computer Networks
Information Security
Compiler Construction
Design and Analysis of Algorithms
Probability and Statistics
Linear Algebra
English · Professional working proficiencyUrdu · Native or bilingual proficiency

Certifications

Udemy

FastAPI: The Complete Course 2025 (Beginner + Advanced)

31 Oct 2025 · 21.5 hours

Udemy

Microsoft Power BI Desktop for Business Intelligence

31 Oct 2025 · 17 hours

Udemy

Data Analyst in Python, Tableau, SQL and ChatGPT with Projects

31 Oct 2025 · 21.5 hours

DataCamp

Understanding Data Science

07 Nov 2025 · 2 hours

DataCamp

Introduction to MongoDB in Python

09 Nov 2025 · 3 hours

Cisco Networking Academy

CCNA: Introduction to Networks

08 Jan 2026

06 · Contact

Let's measure what matters.

Open to AI safety and LLM evaluation research internships, plus backend and data engineering roles. Based in Karachi, Pakistan, working remotely.

Muhammadroshaan1127@gmail.com
GitHubLinkedInEmailRésumé

© 2026 Muhammad Roshaan · Built with Next.js, Tailwind & Framer Motion