Skip to main content Link Menu Expand (external link) Document Search Copy Copied
estela Documentation

estela Resources Appendix

estela needs the following resources to run:

  • Container Orchestrator System:

    This system is used to manage the execution of the scrapy jobs, as well as the management of the services and deployments of the API module and the Queuing module. estela was thought to work with Kubernetes, but any other orchestrator can be used (soon). The system can be managed by a cloud provider, e.g., EKS, or by your local computer, e.g., minikube.

  • SQL Relational Database:

    This database is used to store the metadata of the API module. The default relational database management system is MySQL, but any other manager can be used (soon). The database can be managed by a cloud provider, e.g., RDS, or by your local computer.

  • Document Oriented Database:

    This database is used to store all the data collected from the spiders. The default database management system is MongoDB, but any other manager can be used (soon). The database can be managed by a cloud provider, e.g., MongoDB Atlas, or by your local computer.

  • Docker Container Registry:

    This registry is used to store the Docker images of the API module and the Queuing module. The registry can be managed by a cloud provider, e.g., ECR, or by your local computer.

  • Object Storage Service:

    This storage is used to store the projects deployed from the estela CLI. The storage can be managed by a cloud provider, e.g., S3, or by your local computer, e.g., MinIO.

  • Queuing Platform:

    This platform controls real-time data feeds in a producer-consumer architecture. The default platform is Kafka, but any other platform can be used (soon). The queuing service can be deployed within the container orchestrator system, or in your local computer.

  • SMTP Email Server:

    This server is used to send confirmation emails after creating an account through the web interface. The SMTP server can be dedicated, e.g., SES; or public, e.g., Google SMTP server.

Issues during the process

In case you encounter some issues during the installation process, check out the estela Installation section in our FAQ.