Job Title: Software Engineer (Gen AI Solutions)

Location: Austin, TX

Duration: 14 Months + Extension

Bill Rate: $76/hour (C2C)

Job Type: W-2, 1099, C2C

Client: To Be Discussed Later

Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Job Description :

We need an experienced software engineer to help design, build, and optimize production LLM inference services that power GenAI features for internal and customer-facing applications. The engineer will work with our ML, platform, and product teams to make model serving faster, cheaper, and more reliable at scale.

Scope of work :

• Design, deploy, and maintain scalable inference services for open-weight and hosted LLMs on Kubernetes-based infrastructure.
• Profile and reduce end-to-end latency (time to first token, tokens per second) and cost per request.
• Apply model and serving optimizations, then benchmark the trade-offs between quality, latency, and throughput.
• Build and maintain integrations with external model APIs and internal orchestration frameworks, including routing, fallback, retries, and rate-limit handling.
• Add observability across the request path, covering logging, metrics, tracing, and dashboards for request volume, downstream call ratios, errors, and token usage.
• Set up evaluation and regression pipelines so model or serving changes don't degrade output quality.
• Do capacity planning and autoscaling, and track GPU utilization and cost.
• Write technical design docs and runbooks, and hand knowledge over to the internal team.

Expected deliverables :

• Production-ready inference service(s) with CI/CD and infrastructure as code
• Latency and cost benchmark reports, measured before and after optimization
• Monitoring dashboards and alerting
• Architecture docs, runbooks, and handover sessions

Required :

• Software engineering experience, including 2+ years on ML/LLM systems in production
• Strong Python, plus working knowledge of Go, C++, or Rust for performance-critical paths
• Hands-on experience with LLM serving frameworks: vLLM, TensorRT-LLM, TGI, SGLang, or Triton Inference Server
• Inference optimization techniques:
• Quantization (INT8/INT4, FP8, AWQ, GPTQ)
• KV-cache management, paged attention, prefix caching
• Continuous/dynamic batching and speculative decoding
• Tensor and pipeline parallelism
• Experience with PyTorch and the Hugging Face ecosystem
• Kubernetes, Docker, and GPU scheduling (NVIDIA GPU operator, node pools, autoscaling)
• Integration with hosted LLM APIs (OpenAI, Gemini, Anthropic), including streaming, retries, and rate limiting
• Observability tooling: Prometheus/Grafana, Splunk, Datadog, OpenTelemetry
• Solid grounding in distributed systems, API design (REST/gRPC), and performance profiling

Preferred : 

• GenAI orchestration frameworks: Genkit, LangChain, LlamaIndex
• CUDA, Triton kernels, FlashAttention, or other low-level GPU optimization
• Model compilation and runtimes: ONNX Runtime, TorchScript/torch.compile, Core ML
• RAG pipelines and vector databases (FAISS, Milvus, pgvector)
• LLM evaluation frameworks and building automated quality and regression benchmarks
• Cost modeling and FinOps for GPU workloads
• Infrastructure as code (Terraform, Helm) and CI/CD (GitHub Actions, Jenkins)
• Contributions to open-source inference projects, or published work in ML systems

Soft skills :

• Can turn ambiguous requirements into concrete technical plans and trade-off proposals
• Communicates clearly in writing (design docs, benchmark reports)
• Works well with cross-functional teams of engineers and PMs.