lakehq_sail
by lakehqLakeSail Computation Framework
Overview
LakeSail is a computation framework designed to unify batch processing, stream processing, and compute-intensive (AI) workloads. It provides a drop-in replacement for Spark SQL and the Spark DataFrame API, enabling seamless integration in both single-host and distributed environments.
Key Features
- Unified Processing: Combines batch, stream, and AI workloads into a single framework.
- Spark Compatibility: Fully compatible with Spark SQL and DataFrame API.
- Distributed Processing: Supports distributed processing in Kubernetes clusters.
- MCP Server Integration: Enhances data analytics for both LLM agents and humans.
Installation
Sail is available as a Python package on PyPI. Install it using pip:
pip install "pysail[spark]"
For better performance, you can also build Sail from source. Refer to the Installation Guide for detailed instructions.
Getting Started
Starting the Sail Server
You can start the Sail server using one of the following methods:
Command Line Interface
sail spark server --port 50051
Python API
from pysail.spark import SparkConnectServer
server = SparkConnectServer(port=50051)
server.start(background=False)
Kubernetes Deployment
Deploy Sail on Kubernetes for distributed processing. Follow the Kubernetes Deployment Guide for setup instructions.
kubectl apply -f sail.yaml
kubectl -n sail port-forward service/sail-spark-server 50051:50051
Connecting to the Sail Server
Once the server is running, connect to it using PySpark:
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://localhost:50051").getOrCreate()
spark.sql("SELECT 1 + 1").show()
For more details, refer to the Getting Started Guide.
Documentation
The latest documentation for Sail can be found here.
Further Reading
- Supercharge Spark: Quadruple Speed, Cut Costs by 94% - Benchmark results comparing Sail with Spark.
- Sail 0.2 and the Future of Distributed Processing - Insights into Sail's distributed processing architecture.
Contributing
Contributions are welcome! Submit GitHub issues for bug reports and feature requests. Join GitHub discussions to ask questions or share ideas. For code changes, refer to the Development Guide.
Support
LakeSail offers enterprise support for Sail. Contact us for more information.
About
LakeSail's mission is to unify batch processing, stream processing, and compute-intensive (AI) workloads. Learn more at lakesail.com.
License
Sail is licensed under the Apache-2.0 license.
Topics
Resources
Activity
Contributors
Languages
- Rust: 94.6%
- Python: 4.5%
- Other: 0.9%
Invite Gmail Contacts
