A systematic literature review of mining weak signals and trends for corporate foresight

Due to the ever-growing amount of data, computer-aided methods and systems to detect weak signals and trends for corporate foresight are in increasing demand. To this day, many papers on this topic have been published. However, research so far has only dealt with specific aspects, but it has failed to provide a comprehensive overview of the research domain. In this paper, we conduct a systematic literature review to organize existing insights and knowledge. The 91 relevant papers, published between 1997 and 2017, are analyzed for their distribution over time and research outlets. Classifying them by their distinct properties, we study the data sources exploited and the data mining techniques applied. We also consider eight different purposes of analysis, namely weak signals and trends concerning political, economic, social and technological factors. The results of our systematic review show that the research domain has indeed been attracting growing attention over time. Furthermore, we observe a great variety of data mining and visualization techniques, and present insights on the efficacy and effectiveness of the data mining techniques applied. Our results reveal that a stronger emphasis on search strategies, data quality and automation is required to greatly reduce the human actor bias in the early stages of the corporate foresight process, thus supporting human experts more effectively in later stages such as strategic decision making and implementation. Moreover, systems for detecting weak signals and trends need to be able to learn and accumulate knowledge over time, attaining a holistic view on weak signals and trends, and incorporating multiple source types to provide a solid foundation for strategic decision making. The findings presented in this paper point to future research opportunities, and they can help practitioners decide which sources to exploit and which data mining techniques to apply when trying to detect weak signals and trends.

[1]  William M. Pottenger,et al.  A Survey of Emerging Trend Detection in Textual Data Mining , 2004 .

[2]  J. Leker,et al.  Patent indicators for monitoring convergence - examples from NFF and ICT , 2011 .

[3]  Tuomo Kuosa,et al.  Futures signals sense-making framework (FSSF): A start-up tool to analyse and categorise weak signals, wild cards, drivers, trends and other types of information , 2010 .

[4]  Yi Zhang,et al.  Tracing Technology Evolution Pathways by Combining Tech Mining and Patent Citation Analysis , 2015 .

[5]  H. Russell Bernard,et al.  Social Research Methods: Qualitative and Quantitative Approaches , 2000 .

[6]  Noga Alon,et al.  The Probabilistic Method , 2015, Fundamentals of Ramsey Theory.

[7]  Ophir Frieder,et al.  A framework for detecting public health trends with Twitter , 2013, 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2013).

[8]  Byungun Yoon,et al.  Development of patent roadmap based on technology roadmap by analyzing patterns of patent development , 2015 .

[9]  Frans Coenen,et al.  Trend mining in social networks: from trend identification to visualization , 2014, Expert Syst. J. Knowl. Eng..

[10]  Wei Lu,et al.  Analyzing evolution of research topics with NEViewer: a new method based on dynamic co-word networks , 2014, Scientometrics.

[11]  Dirk Thorleuchter,et al.  Semantic weak signal tracing , 2014, Expert Syst. Appl..

[12]  Hyeonju Seol,et al.  Identifying technological opportunities using the novelty detection technique: a case of laser technology in semiconductor manufacturing , 2013, Technol. Anal. Strateg. Manag..

[13]  Cl Nwakwuo,et al.  The influence of information and communications technology (ICT) on information services delivery in academic libraries in Imo State, in Nigeria. , 2014 .

[14]  R. Rohrbeck,et al.  Environmental Scanning, Futures Research, Strategic Foresight and Organizational Future Orientation: A Review, Integration, and Future Research Directions , 2012 .

[15]  M. Shamim Hossain,et al.  Cross-Platform Emerging Topic Detection and Elaboration from Multimedia Streams , 2015, ACM Trans. Multim. Comput. Commun. Appl..

[16]  Seungmin Rho,et al.  TwitterTrends: a spatio-temporal trend detection and related keywords recommendation scheme , 2013, Multimedia Systems.

[17]  B. U. Kannappanavar,et al.  Information and Knowledge Management , 2007 .

[18]  Luigi Di Caro,et al.  Personalized emerging topic detection based on a term aging model , 2013, ACM Trans. Intell. Syst. Technol..

[19]  Seong Joon Yoo,et al.  Hot topic detection and technology trend tracking for patents utilizing term frequency and proportional document frequency and semantic information , 2016, 2016 International Conference on Big Data and Smart Computing (BigComp).

[20]  Ayse Basar Bener,et al.  Mining trends and patterns of software vulnerabilities , 2016, J. Syst. Softw..

[21]  Sunghae Jun,et al.  Patent Management for Technology Forecasting: A Case Study of the Bio-Industry , 2012 .

[22]  Tae-Eung Sung,et al.  Detection and Analysis of Trend Topics for Global Scientific Literature Using Feature Selection Based on Gini-Index , 2011, 2011 IEEE 23rd International Conference on Tools with Artificial Intelligence.

[23]  Sunghae Jun,et al.  Technology Forecasting using Matrix Map and Patent Clustering , 2012, Ind. Manag. Data Syst..

[24]  Dirk Thorleuchter,et al.  Weak signal identification with semantic web mining , 2013, Expert Syst. Appl..

[25]  Marco A. Palomino,et al.  Optimising web-based information retrieval methods for horizon scanning using relevance feedback , 2013, 2013 Federated Conference on Computer Science and Information Systems.

[26]  Jun Wang,et al.  Mean-Variance Analysis: A New Document Ranking Theory in Information Retrieval , 2009, ECIR.

[27]  Siu Cheung Hui,et al.  Web Mining for Identifying Research Trends , 2003, ICADL.

[28]  Yongtae Park,et al.  Monitoring trends of technological changes based on the dynamic patent lattice: A modified formal concept analysis approach , 2011 .

[29]  Tom Brier,et al.  Communications of the Association for Information Systems , 1999 .

[30]  Olfa Nasraoui,et al.  A framework for mining evolving trends in Web data streams using dynamic learning and retrospective validation , 2006, Comput. Networks.

[31]  Ming-Yeu Wang,et al.  Identifying Technology Trends for R&D Planning Using TRIZ and Text Mining , 2010 .

[32]  Matias Welling Flensborg.  REVIEW: "Competitive intelligence and patent analysis in drug discovery: Mining the competitive knowledge bases and patents" , 2008 .

[33]  Pengzhu Zhang,et al.  Health-Related Hot Topic Detection in Online Communities Using Text Clustering , 2013, PloS one.

[34]  Ying Zhu,et al.  Detecting Hotspot Information Using Multi-Attribute Based Topic Model , 2015, PloS one.

[35]  Aditi Sharan,et al.  A Framework for Automatic Query Expansion , 2010, WISM.

[36]  Martin G. Moehrle,et al.  Anticipating industry convergence: semantic analyses vs IPC co-classification analyses of patents , 2013 .

[37]  Daniel A. Keim,et al.  State-of-the-Art Report of Visual Analysis for Event Detection in Text Data Streams , 2014, EuroVis.

[38]  Thomas Mohr,et al.  Environmental Scanning Systems: State Of The Art And First Instantiation , 2011, PACIS.

[39]  Christian Bauckhage,et al.  Detecting Trends in Social Bookmarking Systems: A del.icio.us Endeavor , 2010, Int. J. Data Warehous. Min..

[40]  Bhaskar Mukherjee,et al.  1 Information Technology and Knowledge Management , 2019 .

[41]  Farshad Madani.  ‘Technology Mining’ bibliometrics analysis: applying network analysis and cluster analysis , 2015, Scientometrics.

[42]  Gayle H. McColloch,et al.  Coal in West Virginia: Geology and Current Mining Trends: ABSTRACT , 1980 .

[43]  Chao-Chan Wu,et al.  Using patent analyses to monitor the technological trends in an emerging field of technology: a case of carbon nanotube field emission display , 2009, Scientometrics.

[44]  Yi-Ning Tu,et al.  Constructing conceptual trajectory maps to trace the development of research fields , 2016, J. Assoc. Inf. Sci. Technol..

[45]  Hongfei Lin,et al.  Detection and Extraction of Hot Topics on Chinese Microblogs , 2015, Cognitive Computation.

[46]  Martin G. Moehrle,et al.  A new instrument for technology monitoring: novelty in patents measured by semantic patent analysis , 2012, Scientometrics.

[47]  Stuart E. Madnick,et al.  Semantic distances for technology landscape visualization , 2012, Journal of Intelligent Information Systems.

[48]  G. Aghila,et al.  Text Mining Process, Techniques and Tools : an Overview , 2010 .

[49]  Sungjoo Lee,et al.  An approach to discovering new technology opportunities: Keyword-based patent map approach , 2009 .

[50]  Harry Commandeur,et al.  Food-Pharma Convergence in Medical Nutrition– Best of Both Worlds? , 2013, PloS one.

[51]  Seonho Kim,et al.  NEST: A quantitative model for detecting emerging trends using a global monitoring expert network and Bayesian network , 2013 .

[52]  Gregorio González-Alcaide,et al.  Bibliometric indicators to identify emerging research fields: publications on mass gatherings , 2016, Scientometrics.

[53]  Federico Caviggioli,et al.  Technology fusion: Identification and analysis of the drivers of technology convergence using patent data , 2016 .

[54]  Third International Conference on Knowledge Discovery and Data Mining, WKDD 2010, Phuket, Thailand, 9-10 January 2010 , 2010, WKDD.

[55]  Alan L. Porter,et al.  Identification of technology development trends based on subject–action–object analysis: The case of dye-sensitized solar cells , 2015 .

[56]  Pei-Chun Lee,et al.  Integrated methodologies for mapping and forecasting science and technology trends: A case of etching technology , 2010, PICMET 2010 TECHNOLOGY MANAGEMENT FOR GLOBAL ECONOMIC GROWTH.

[57]  Gilda Massari Coelho,et al.  Text mining as a valuable tool in foresight exercises: A study on nanotechnology , 2006 .

[58]  Eva Blomqvist,et al.  The use of Semantic Web technologies for decision support - a survey , 2014, Semantic Web.

[59]  H. Ansoff,et al.  Managing Strategic Surprise by Response to Weak Signals , 1975 .

[60]  Alan L. Porter,et al.  Analyzing patent topical information to identify technology pathways and potential opportunities , 2014, Scientometrics.

[61]  Min Song,et al.  Analyzing the Political Landscape of 2012 Korean Presidential Election in Twitter , 2014, IEEE Intelligent Systems.

[62]  Myong Kee Jeong,et al.  Patent Clustering and Outlier Ranking Methodologies for Attributed Patent Citation Networks for Technology Opportunity Discovery , 2016, IEEE Transactions on Engineering Management.

[63]  Bruno Agard,et al.  Discovering and assessing fields of expertise in nanomedicine: a patent co-citation network perspective , 2012, Scientometrics.

[64]  Mitsuru Ishizuka,et al.  Emerging topic tracking system in WWW , 2006, Knowl. Based Syst..

[65]  Wolfgang Glänzel,et al.  Using ‘core documents’ for detecting and labelling new emerging topics , 2011, Scientometrics.

[66]  Ichiro Sakata,et al.  Detecting emerging research fronts in regenerative medicine by the citation network analysis of scientific publications , 2011 .

[67]  Xiuzhen Zhang,et al.  A probabilistic method for emerging topic tracking in Microblog stream , 2016, World Wide Web.

[68]  Vasilis Stavrou,et al.  Data Mining for Knowledge Discovery , 2015 .

[69]  Heiko A. von der Gracht,et al.  The influence of information and communication technology (ICT) on future foresight processes — Results from a Delphi survey , 2014 .

[70]  Yi-Ning Tu,et al.  Indices of novelty for emerging topic detection , 2012, Inf. Process. Manag..

[71]  Hyeokseong Lee,et al.  Dynamic Patterns of Industry Convergence: Evidence from a Large Amount of Unstructured Data , 2015 .

[72]  Woo Hyoung Lee,et al.  How to identify emerging research fields using scientometrics: An example in the field of Information Security , 2008, Scientometrics.

[73]  Hongjun Lu,et al.  Knowledge discovery and data mining , 1998, Knowl. Based Syst..

[74]  Duen-Ren Liu,et al.  Discovering competitive intelligence by mining changes in patent trends , 2010, Expert Syst. Appl..

[75]  Chaomei Chen,et al.  Web site design with the patron in mind: A step-by-step guide for libraries , 2006 .

[76]  Heiko A. Gracht,et al.  Corporate foresight and innovation management: A portfolio-approach in evaluating organizational development , 2010 .

[77]  Stephen E. Robertson,et al.  Understanding inverse document frequency: on theoretical arguments for IDF , 2004, J. Documentation.

[78]  Chaomei Chen,et al.  CiteSpace II: Detecting and visualizing emerging trends and transient patterns in scientific literature , 2006, J. Assoc. Inf. Sci. Technol..

[79]  Xiaolong Wang,et al.  Online topic detection and tracking of financial news based on hierarchical clustering , 2010, 2010 International Conference on Machine Learning and Cybernetics.

[80]  Harukazu Igarashi,et al.  INTERNATIONAL JOURNAL OF ARTIFICIAL INTELLIGENCE AND EXPERT SYSTEMS (IJAE) , 2012 .

[81]  Yongtae Park,et al.  Development of New Technology Forecasting Algorithm: Hybrid Approach for Morphology Analysis and Conjoint Analysis of Patent Information , 2007, IEEE Transactions on Engineering Management.

[82]  Yongtae Park,et al.  Technology opportunity identification customized to the technological capability of SMEs through two-stage patent analysis , 2013, Scientometrics.

[83]  Zhong Liu,et al.  Mining research trends with anomaly detection models: the case of social computing research , 2015, Scientometrics.

[84]  ChengXiang Zhai,et al.  Discovering evolutionary theme patterns from text: an exploration of temporal text mining , 2005, KDD '05.

[85]  Jan Oliver Schwarz,et al.  Pitfalls in implementing a strategic early warning system , 2005 .

[86]  Mu-Hsuan Huang,et al.  Detecting research fronts in OLED field using bibliographic coupling with sliding window , 2013, Scientometrics.

[87]  Ramakrishnan Srikant,et al.  Discovering Trends in Text Databases , 1997, KDD.

[88]  A. E. A. da Silva,et al.  A Clustering Method for Weak Signals to Support Anticipative Intelligence , 2015 .

[89]  Mohamed Medhat Gaber,et al.  TRCM: A Methodology for Temporal Analysis of Evolving Concepts in Twitter , 2013, ICAISC.

[90]  Andrew McCallum,et al.  Topics over time: a non-Markov continuous-time model of topical trends , 2006, KDD '06.

[91]  Stuart E. Madnick,et al.  A framework for technology forecasting and visualization , 2009, 2009 International Conference on Innovations in Information Technology (IIT).

[92]  Yiannis Kompatsiaris,et al.  Sensing Trending Topics in Twitter , 2013, IEEE Transactions on Multimedia.

[93]  Yun Chi,et al.  Eigen-trend: trend analysis in the blogosphere based on singular value decompositions , 2006, CIKM '06.

[94]  Shintaro Okazaki,et al.  Combining social-based data mining techniques to extract collective trends from twitter , 2014 .

[95]  Jie Lu,et al.  A patent time series processing component for technology intelligence by trend identification functionality , 2014, Neural Computing and Applications.

[96]  Jörg H. Mayer,et al.  Improving the Applicability of Environmental Scanning Systems: State of the Art and Future Research , 2011, Governance and Sustainability in Information Systems.

[97]  Ozcan Saritas,et al.  A methodology for technology trend monitoring: the case of semantic technologies , 2016, Scientometrics.

[98]  Roberto V. Zicari,et al.  PoliTwi: Early detection of emerging political topics on twitter and the impact on concept-level sentiment analysis , 2014, Knowl. Based Syst..

[99]  Sunghae Jun,et al.  Methodology of technological evolution for three-dimensional printing , 2016, Ind. Manag. Data Syst..

[100]  Myong Kee Jeong,et al.  Inter-cluster connectivity analysis for technology opportunity discovery , 2014, Scientometrics.

[101]  Olesya Mryglod,et al.  Quantifying the evolution of a scientific topic: reaction of the academic community to the Chornobyl disaster , 2015, Scientometrics.

[102]  Wolfgang Gaul,et al.  Evaluation of the evolution of relationships between topics over time , 2017, Adv. Data Anal. Classif..

[103]  Myra Spiliopoulou,et al.  Discovering Emerging Topics in Unlabelled Text Collections , 2006, ADBIS.

[104]  Samee U. Khan,et al.  A literature review on the state-of-the-art in patent analysis , 2014 .

[105]  Kwangsoo Kim,et al.  Detecting signals of new technological opportunities using semantic patent analysis and outlier detection , 2011, Scientometrics.

[106]  Abdul-Aziz Rashid Al-Azmi.  Data, text and web mining for business intelligence: a survey , 2013, ArXiv.

[107]  Jack E. Smith,et al.  The Big Picture – trends, drivers, wild cards, discontinuities and weak signals , 2011 .

[108]  Kwangsoo Kim,et al.  SAO network analysis of patents for technology trends identification: a case study of polymer electrolyte membrane technology in proton exchange membrane fuel cells , 2011, Scientometrics.

[109]  Douglas Henrique Milanez,et al.  Patents in nanotechnology: an analysis using macro-indicators and forecasting curves , 2014, Scientometrics.

[110]  Shusaku Tsumoto,et al.  Trend detection from large text data , 2010, 2010 IEEE International Conference on Systems, Man and Cybernetics.

[111]  Ryota Tomioka,et al.  Discovering Emerging Topics in Social Streams via Link-Anomaly Detection , 2014, IEEE Transactions on Knowledge and Data Engineering.

[112]  Xuwei Pan,et al.  Identifying digital traces for business marketing through topic probabilistic model , 2015, Technol. Anal. Strateg. Manag..

[113]  Katy Börner,et al.  Mixed-indicators model for identifying emerging research areas , 2011, Scientometrics.

[114]  Amy J. C. Trappey,et al.  Using patent data for technology forecasting: China RFID patent analysis , 2011, Adv. Eng. Informatics.

[115]  Manuel C Peitsch,et al.  Competitive intelligence and patent analysis in drug discovery. , 2005, Drug discovery today. Technologies.

[116]  Heeyong Noh,et al.  Identifying emerging core technologies for the future , 2016 .

[117]  Yunming Ye,et al.  Detecting hot topics from Twitter: A multiview approach , 2014, J. Inf. Sci..

[118]  Elina Hiltunen,et al.  The future sign and its three dimensions , 2008 .

[119]  C. Lee Giles,et al.  Topic and Trend Detection in Text Collections Using Latent Dirichlet Allocation , 2009, ECIR.

[120]  Duen-Ren Liu,et al.  Mining the change of event trends for decision support in environmental scanning , 2009, Expert Syst. Appl..

[121]  Heinrich Arnold,et al.  IT Tools for Foresight: The Integrated Insight and Response System of Deutsche Telekom Innovation Laboratories , 2013 .

[122]  Christoph Meinel,et al.  Identify Emergent Trends Based on the Blogosphere , 2013, 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT).

[123]  Lyle Ungar,et al.  Discovery of significant emerging trends , 2010, KDD.

[124]  J. Youtie,et al.  Refining search terms for nanotechnology , 2008 .

[125]  Juan D. Velásquez,et al.  Detecting trends on the Web: A multidisciplinary approach , 2014, Inf. Fusion.

[126]  Pan Jun Kim,et al.  Domain analysis with text mining: Analysis of digital library research trends using profiling methods , 2010, J. Inf. Sci..

[127]  Eitan Altman,et al.  Trend detection in social networks using Hawkes processes , 2015, 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[128]  Chia-Hui Chang,et al.  Exploring Evolutionary Technical Trends from Academic Research Papers , 2008, 2008 The Eighth IAPR International Workshop on Document Analysis Systems.

[129]  Stijn Viaene,et al.  Linking technology intelligence to open innovation , 2010 .

[130]  Padhraic Smyth,et al.  From Data Mining to Knowledge Discovery in Databases , 1996, AI Mag..

[131]  Björn Niehaves,et al.  Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information Systems Research , 2015, Commun. Assoc. Inf. Syst..

[132]  Jinhyung Kim,et al.  Technology trends analysis and forecasting application based on decision tree and statistical feature analysis , 2012, Expert Syst. Appl..

[133]  Jan W Kantelhardt,et al.  The Detection of Emerging Trends Using Wikipedia Traffic Data and Context Networks , 2015, PloS one.