The authors describe a three-dimensional opto-electronic neural system that uses a hybrid silicon VLSI-based opto-electronic integrated circuit (OEIC) technology to implement the neurons and their associated synapses and fixed, free-space optical diffractive elements to interconnect the neurons. The architecture provides full connectivity between neurons, flexible functionality of neurons and synapses, accurate electronic fan-in, and biologically inspired dendritic-type fan-in processing. The system requirements for neural network implementations are addressed. The architecture is described, and the technological feasibility of the system is discussed. Optimal data encoding methods are proposed. Low-area circuit designs with learning capabilities that achieve high linear dynamic range are presented.