Scalable multicore systems require a cache-coherency scheme that can grow with the system, yet is energy efficient and can adapt to many patterns of system usage. The Distributed Discretionary Directories and Data (4D) cache-coherency protocol is a fully distributed protocol for shared caches organized in a tree-like hierarchy. 4D guarantees deadlock freedom and sequential consistency (or other memory models) while allowing each cache node to retain or discard directory information and data at any time depending on the operating context.
Thus, 4D provides a robust and flexible mechanism that can support a large set of policies and optimizations.

Zettabricks is a novel language, compiler and runtime system that eliminates the programmer's burden of writing efficient parallel code targeted for a particular multi-core architecture. The system saves programmer effort and reduces the inefficiency of poorly targeted code allowing energy savings and better program performance. The programmer exposes algorithmic choices to the compiler via the Zettabricks language. The Zettabricks compiler and autotuner then exploit both offline and online machine learning techniques to compose a hybrid algorithm for optimal execution on a target architecture.

SEFOS is a new self-aware, factored OS designed from the ground up for scalability, dependability, and adaptability and is targeted at 1000+ core systems.  It ties all of the other Angstrom hardware and software components together and helps address all four of the key extreme-scale challenges of programmability, scalability, dependability and cyber resilience, and energy efficiency.

Extreme-scale computers will contain 1000s of cores. Rather than run application threads on every available core, an alternate approach is to employ some cores that would otherwise be used inefficiently to run ``helper threads.'' Helper threads assist the application's compute threads in some way. One example use of helper threads is to trigger in advance long-latency cache misses or branch mispredictions that would normally stall compute threads, thus improving performance and power efficiency. Our research investigates how to construct helper threads, how to support them on extremely low-power cores (e.g., Angstrom's partner cores), and how to coordinate both compute and helper threads in 1000-core processors.