Performance Optimization Through Distributed Data Processing and Microservices

Abstract

Microservice decomposition and distributed data processing are each mature engineering disciplines, but they optimize for different, and sometimes conflicting, objectives. Microservice decomposition literature evaluates a service boundary primarily by software-engineering criteria — cohesion, coupling, independent deployability, team ownership — while distributed data processing performance is dominated by data-locality concerns largely orthogonal to those criteria: how data is partitioned, how much of it must be shuffled across the network to complete a computation, and how skewed that data is across partitions. A service boundary that scores well on cohesion and deployability can still force a distributed processing workload to shuffle a large volume of data across exactly that boundary on every request, and a container-orchestration platform that schedules service instances without awareness of this data-movement cost can place communicating services far apart on the underlying cluster even when doing so is avoidable. This paper proposes LOCUS (Locality-Optimized Coordination for Unified microServices), an architecture in which a data-locality-aware decomposition layer, a partition-aware distributed processing layer, and a locality-aware scheduling layer are coordinated by a shared locality cost model that estimates the data-movement cost of a candidate service boundary, data partitioning scheme, or scheduling placement before it is committed to, rather than optimizing each decision against its own narrower criteria in isolation. Following a design-science research approach, LOCUS is illustrated, not empirically validated, through a comparative capability table, a short case scenario, and two illustrative figures, with every quantitative value explicitly labeled as an illustrative estimate rather than measured production data. The paper situates LOCUS against the literature on microservice decomposition and design patterns, distributed data processing and data skew, stream-processing optimization, microservice workload characterization, and container orchestration.