DGrid AI tackles the decentralized reward problem with reference-free scoring

The current standard for decentralized inference relies on semantic similarity, measuring the distance between model outputs and known answers. This approach collapses in real-world scenarios where user queries are open-ended. DGrid’s new research moves away from this dependency by training three specialized judges—TextCNN, MiniLM, and DeBERTa—to assign scores from 0 to 10 based on inherent response quality rather than external benchmarks.

Performance metrics indicate a significant shift in accuracy. The DeBERTa-based judge reached a 0.747 Pearson correlation against ground-truth proxies, outperforming previous reference-based frameworks which peaked at 0.647. The researchers achieved this by pre-training on the UltraFeedback dataset before fine-tuning on the network’s specific task distribution. To manage computational costs, the team implemented a cascading pipeline that routes queries through lighter models first, escalating to heavier evaluators only when ambiguity arises. This configuration can reduce evaluation expenses by up to 72.7%.

Despite these gains, the system faces uneven performance across different task types. While question-answering accuracy is robust, correlation drops to 0.199 for summarization tasks. The authors attribute this discrepancy to the training data’s reliance on word overlap, which serves as a poor proxy for summarization quality. Rather than masking these limitations, the paper frames them as the primary open problems, signaling an iterative approach to building production-ready infrastructure for decentralized networks.