The embracing face has become synonymous with advances in AI on a large scale. With over 4 million builders deploying their models in their hubs, the rapid growth of the platform has forced them to rethink how secret sensitive data can be managed.
Last year, the engineering team set out to improve its handling of secrecy and credentials. After evaluating tools like Hashicorp Vault, I finally chose Infisical.
This case study details the transition to Infis, explains how powerful features were integrated, and highlights how engineers can operate more efficiently and safely.
background
As Face’s infrastructure evolved from an AWS-only setup to a multi-cloud environment with Azure and GCP, engineering teams needed a more agile, secure and centralized way to manage secrets. Instead of reworking legacy systems or adopting heavyweight solutions like Hashicop Vault, developer-friendly workflows, multi-cloud abstractions and robust security features have made them irrational.
The key challenges they faced were:
Increased risk of “secret sprawl” due to inconsistent management of the entire environment. The complex team-grossed permission management requires tight, role-based access control (RBAC) integrated with your organization’s SSO (OKTA). Traditional .ENV files can undermine both security and developer productivity, and regional development difficulties. The burden of secret revolving became painfully clear after a security incident that exposed the qualification.
Additionally, the team needed a solution that adhered to infrastructure practices as code, supported project-by-project secret management, and provided a smooth balance between automation and manual control during deployment.
implementation
Infisical’s flexible architecture was the ideal solution. The engineering team seized the opportunity to revisit the internal project structure and split the project into different infrastructure and application domains. This allowed them to implement a clearer separation of concerns and standardize secret rotation practices. This has been given priority in the wake of recent security incidents.
By leveraging Terraform, which was previously used to create Kubernetes secrets from AWS configurations, they found the migration to Infisical Kubernetes operators to be extremely smooth. This integration allows for improved security while standardizing secret management across all environments.
Kubernetes Integration
Kubernetes is the heart of embracing Face’s production environment, and Infisical’s Kubernetes operators have been instrumental in automating secret updates. The operator continuously monitors Infisical’s secret changes and ensures that these updates are propagated to the corresponding Kubernetes object. Each time a change is detected, the dependent deployment can be automatically reloaded, ensuring that the container is always running with the latest secrets.
example:
Applications running on Kubernetes require new secrets. Secrets can be created via Infisical’s CLI or the web UI, and developers create InfisicalSecret resources in Kubernetes and specify which secrets should be synchronized from Infis.
Apiversion: inisic.com/v1alpha1
Kindness: InisicalSecret
Metadata:
name: My App Celebrity
Namespace: production
specification:
InisicalSecretid: “123E4567-E89B-12D3-A456-426614174000”
TargetSecretname: “My-App-K8S-Secret”
Once CRD is applied, the Infisical Operator will continuously monitor updates. If a change is detected falsely, the operator automatically updates the kubernetes secret (my-app-k8s-secret).
Better yet, application deployment refers to My-App-K8S-Secret as an environment variable source or mount volume, allowing operators to automatically trigger the container when the secret changes.
In reality, face engineers prefer to wait for manual relocation despite their ability to automatically trigger container reboots. This decision was driven by the need to accurately control deployment, especially when there are heavy traffic (more than 10 million requests per minute), when there are many replicas involved.
Local Development
For local development, Infisical’s CLI streamlines workflows by injecting secrets directly into the development environment. This removes the need for unstable local .ENV files, aligns production standards for local configurations, and removes reductions in onboarding friction.
Security and Access Management
Improved security forms the backbone of this transition. By integrating Infisical with existing identity providers such as OKTA, Face has established a fine-grained RBAC system. Permissions are automatically mapped from the OKTA group to ensure that developers retain control over the project, and front-end and back-end teams receive appropriately restricted read or write access.
Additionally, the secret sharing feature allows for secure credentials to be shared among Face’s ML/AI researchers. The centralized INFISICAL platform simplifies auditing and management of secret rotations. This is a need highlighted by previous security incidents.
CI/CD and infrastructure integration
Seamless integration with the CI/CD pipeline further enhances the overall security attitude. Inisical was embedded in the deployment pipeline via GitHub actions using OIDC authentication and Terraform integration. By operating self-hosted runners in a secure environment, all deployments comply with production-grade security standards. This integrated approach minimized risk and ensured uniform experience from local development to cloud deployment.
Technical achievements and insights
It has brought about irrational concrete improvements to the management of secrets.
Engineers no longer have to spend valuable time manually configuring environmental secrets. Self-service workflow accelerated the onboarding and daily development cycle. Automated auditing and granular access controls enable rapid incident response and promotes a “left-shift” approach to security. Consistent integration between cloud providers, Kubernetes clusters and CI/CD pipelines eliminates discrepancies in secret management and enhances infrastructure security and reliability.
As Adrien Carreira, head of infrastructure at Hug Face, pointed out:
“INFISICAL provides all the functionality and security settings you need to increase your security attitude and save engineering time. Whether you’re working locally, running a Kubernetes cluster in production, or operating secrets within a CI/CD pipeline, Infisical has a seamless, pre-built workflow.”
Conclusion
Embracing Face’s move to Infisical shows that a technically driven, engineering-centric approach to managing secrets across multiple cloud platforms offers significant benefits. To tackle similar challenges, using INFISICAL is a practical way to work more efficiently while still maintaining strong security.
When a safe pass becomes the easiest pass, teams can focus on building innovative products rather than worrying about secret management.
resource
For teams interested in adopting a similar approach:
This technical case study was adopted from the original case study published at inifisical.com/customers/hugging-face.
